From bc571c8c9046d322d0dd04d3ef5fbd0917b885df Mon Sep 17 00:00:00 2001 From: "Dustin J. Mitchell" Date: Sun, 14 Feb 2010 15:02:32 -0600 Subject: [PATCH 1/6] split documentation into per-chapter files --- .gitignore | 2 + docs/Makefile | 8 +- docs/buildbot.texinfo | 9496 +----------------------------------- docs/buildprocess.texinfo | 2914 +++++++++++ docs/changesources.texinfo | 1139 +++++ docs/cmdline.texinfo | 776 +++ docs/concepts.texinfo | 775 +++ docs/configuration.texinfo | 1529 ++++++ docs/installation.texinfo | 648 +++ docs/introduction.texinfo | 313 ++ docs/resources.texinfo | 9 + docs/status.texinfo | 1054 ++++ docs/version.py | 6 + 13 files changed, 9215 insertions(+), 9454 deletions(-) create mode 100644 docs/buildprocess.texinfo create mode 100644 docs/changesources.texinfo create mode 100644 docs/cmdline.texinfo create mode 100644 docs/concepts.texinfo create mode 100644 docs/configuration.texinfo create mode 100644 docs/installation.texinfo create mode 100644 docs/introduction.texinfo create mode 100644 docs/resources.texinfo create mode 100644 docs/status.texinfo create mode 100644 docs/version.py diff --git a/.gitignore b/.gitignore index fb6e21d4945..17016ae8b55 100644 --- a/.gitignore +++ b/.gitignore @@ -28,3 +28,5 @@ docs/images/status.png .pydevproject .settings twisted/ +docs/buildbot +docs/version.texinfo diff --git a/docs/Makefile b/docs/Makefile index f64c35af620..d4999e45ac2 100644 --- a/docs/Makefile +++ b/docs/Makefile @@ -1,17 +1,23 @@ buildbot.info: buildbot.texinfo + python version.py > version.texinfo makeinfo --fill-column=70 $< buildbot.html: buildbot.texinfo images-png + python version.py > version.texinfo makeinfo --no-split --html $< +buildbot: buildbot.texinfo images-png + python version.py > version.texinfo + makeinfo --html $< + buildbot.ps: buildbot.texinfo images-eps + python version.py > version.texinfo texi2dvi $< dvips buildbot.dvi rm buildbot.aux buildbot.cp buildbot.cps buildbot.fn buildbot.ky buildbot.log buildbot.pg buildbot.toc buildbot.tp buildbot.vr rm buildbot.dvi - .PHONY: images images-png images-eps images: $(MAKE) -C images all diff --git a/docs/buildbot.texinfo b/docs/buildbot.texinfo index a077ce0107c..e8bb598ae5b 100644 --- a/docs/buildbot.texinfo +++ b/docs/buildbot.texinfo @@ -1,7 +1,8 @@ \input texinfo @c -*-texinfo-*- @c %**start of header @setfilename buildbot.info -@settitle BuildBot Manual 0.7.12 +@include version.texinfo +@settitle BuildBot Manual - @value{VERSION} @defcodeindex cs @defcodeindex sl @defcodeindex bf @@ -20,7 +21,7 @@ @c @bcindex : keys that make up BuildmasterConfig @copying -This is the BuildBot manual. +This is the BuildBot manual for Buildbot version @value{VERSION}. Copyright (C) 2005, 2006, 2009, 2010 Brian Warner @@ -32,6 +33,7 @@ provided the copyright notice and this notice are preserved. @titlepage @title BuildBot +@subtitle @value{VERSION} @page @vskip 0pt plus 1filll @insertcopying @@ -49,9461 +51,57 @@ provided the copyright notice and this notice are preserved. @menu * Introduction:: What the BuildBot does. -* Installation:: Creating a buildmaster and buildslaves, - running them. +* Installation:: Installing and setting up a Builbot system * Concepts:: What goes on in the buildbot's little mind. * Configuration:: Controlling the buildbot. * Getting Source Code Changes:: Discovering when to run a build. * Build Process:: Controlling how each build is run. * Status Delivery:: Telling the world about the build's results. -* Command-line tool:: +* Command-line Tool:: The 'buildbot' utility * Resources:: Getting help. -* Developer's Appendix:: -* Index of Useful Classes:: -* Index of master.cfg keys:: -* Index:: Complete index. - -@detailmenu - --- The Detailed Node Listing --- - -Introduction - -* History and Philosophy:: -* System Architecture:: -* Control Flow:: - -System Architecture - -* BuildSlave Connections:: -* Buildmaster Architecture:: -* Status Delivery Architecture:: - -Installation - -* Requirements:: -* Installing the code:: -* Creating a buildmaster:: -* Upgrading an Existing Buildmaster:: -* Creating a buildslave:: -* Launching the daemons:: -* Logfiles:: -* Shutdown:: -* Maintenance:: -* Troubleshooting:: - -Creating a buildslave - -* Buildslave Options:: - -Troubleshooting - -* Starting the buildslave:: -* Connecting to the buildmaster:: -* Forcing Builds:: - -Concepts - -* Version Control Systems:: -* Schedulers:: -* BuildSet:: -* BuildRequest:: -* Builder:: -* Users:: -* Build Properties:: - -Version Control Systems - -* Generalizing VC Systems:: -* Source Tree Specifications:: -* How Different VC Systems Specify Sources:: -* Attributes of Changes:: - -Users - -* Doing Things With Users:: -* Email Addresses:: -* IRC Nicknames:: -* Live Status Clients:: - -Configuration - -* Config File Format:: -* Loading the Config File:: -* Testing the Config File:: -* Defining the Project:: -* Change Sources and Schedulers:: -* Setting the slaveport:: -* Buildslave Specifiers:: -* On-Demand ("Latent") Buildslaves:: -* Defining Global Properties:: -* Defining Builders:: -* Defining Status Targets:: -* Debug options:: - -Change Sources and Schedulers - -* Scheduler Scheduler:: -* AnyBranchScheduler:: -* Dependent Scheduler:: -* Periodic Scheduler:: -* Nightly Scheduler:: -* Try Schedulers:: -* Triggerable Scheduler:: - -Buildslave Specifiers -* When Buildslaves Go Missing:: - -On-Demand ("Latent") Buildslaves -* Amazon Web Services Elastic Compute Cloud ("AWS EC2"):: -* Dangers with Latent Buildslaves:: -* Writing New Latent Buildslaves:: - -Getting Source Code Changes - -* Change Sources:: -* Choosing ChangeSources:: -* CVSToys - PBService:: -* Mail-parsing ChangeSources:: -* PBChangeSource:: -* P4Source:: -* BonsaiPoller:: -* SVNPoller:: -* MercurialHook:: -* Bzr Hook:: -* Bzr Poller:: - -Mail-parsing ChangeSources - -* Subscribing the Buildmaster:: -* Using Maildirs:: -* Parsing Email Change Messages:: - -Parsing Email Change Messages - -* FCMaildirSource:: -* SyncmailMaildirSource:: -* BonsaiMaildirSource:: -* SVNCommitEmailMaildirSource:: - -Build Process - -* Build Steps:: -* Interlocks:: -* Build Factories:: - -Build Steps - -* Common Parameters:: -* Using Build Properties:: -* Source Checkout:: -* ShellCommand:: -* Simple ShellCommand Subclasses:: -* Python BuildSteps:: -* Transferring Files:: -* Steps That Run on the Master:: -* Triggering Schedulers:: -* Writing New BuildSteps:: - -Source Checkout - -* CVS:: -* SVN:: -* Darcs:: -* Mercurial:: -* Arch:: -* Bazaar:: -* Bzr:: -* P4:: -* Git:: - -Simple ShellCommand Subclasses - -* Configure:: -* Compile:: -* Test:: -* TreeSize:: -* PerlModuleTest:: -* Testing with mysql-test-run:: -* SetProperty:: -* SubunitShellCommand:: - -Python BuildSteps - -* BuildEPYDoc:: -* PyFlakes:: -* PyLint:: - -Writing New BuildSteps - -* BuildStep LogFiles:: -* Reading Logfiles:: -* Adding LogObservers:: -* BuildStep URLs:: - -Build Factories - -* BuildStep Objects:: -* BuildFactory:: -* Process-Specific build factories:: - -BuildStep Objects - -* BuildFactory Attributes:: -* Quick builds:: - -BuildFactory - -* BuildFactory Attributes:: -* Quick builds:: - -Process-Specific build factories - -* GNUAutoconf:: -* CPAN:: -* Python distutils:: -* Python/Twisted/trial projects:: - -Status Delivery - -* WebStatus:: -* MailNotifier:: -* IRC Bot:: -* PBListener:: -* Writing New Status Plugins:: - -WebStatus - -* WebStatus Configuration Parameters:: -* Buildbot Web Resources:: -* XMLRPC server:: - -Command-line tool - -* Administrator Tools:: -* Developer Tools:: -* Other Tools:: -* .buildbot config directory:: - -Developer Tools - -* statuslog:: -* statusgui:: -* try:: - -waiting for results - -* try --diff:: - -Other Tools - -* sendchange:: -* debugclient:: - -@end detailmenu -@end menu - -@node Introduction, Installation, Top, Top -@chapter Introduction - -@cindex introduction - -The BuildBot is a system to automate the compile/test cycle required by most -software projects to validate code changes. By automatically rebuilding and -testing the tree each time something has changed, build problems are -pinpointed quickly, before other developers are inconvenienced by the -failure. The guilty developer can be identified and harassed without human -intervention. By running the builds on a variety of platforms, developers -who do not have the facilities to test their changes everywhere before -checkin will at least know shortly afterwards whether they have broken the -build or not. Warning counts, lint checks, image size, compile time, and -other build parameters can be tracked over time, are more visible, and -are therefore easier to improve. - -The overall goal is to reduce tree breakage and provide a platform to -run tests or code-quality checks that are too annoying or pedantic for -any human to waste their time with. Developers get immediate (and -potentially public) feedback about their changes, encouraging them to -be more careful about testing before checkin. - -Features: - -@itemize @bullet -@item -run builds on a variety of slave platforms -@item -arbitrary build process: handles projects using C, Python, whatever -@item -minimal host requirements: python and Twisted -@item -slaves can be behind a firewall if they can still do checkout -@item -status delivery through web page, email, IRC, other protocols -@item -track builds in progress, provide estimated completion time -@item -flexible configuration by subclassing generic build process classes -@item -debug tools to force a new build, submit fake Changes, query slave status -@item -released under the GPL -@end itemize - -@menu -* History and Philosophy:: -* System Architecture:: -* Control Flow:: -@end menu - - -@node History and Philosophy, System Architecture, Introduction, Introduction -@section History and Philosophy - -@cindex Philosophy of operation - -The Buildbot was inspired by a similar project built for a development -team writing a cross-platform embedded system. The various components -of the project were supposed to compile and run on several flavors of -unix (linux, solaris, BSD), but individual developers had their own -preferences and tended to stick to a single platform. From time to -time, incompatibilities would sneak in (some unix platforms want to -use @code{string.h}, some prefer @code{strings.h}), and then the tree -would compile for some developers but not others. The buildbot was -written to automate the human process of walking into the office, -updating a tree, compiling (and discovering the breakage), finding the -developer at fault, and complaining to them about the problem they had -introduced. With multiple platforms it was difficult for developers to -do the right thing (compile their potential change on all platforms); -the buildbot offered a way to help. - -Another problem was when programmers would change the behavior of a -library without warning its users, or change internal aspects that -other code was (unfortunately) depending upon. Adding unit tests to -the codebase helps here: if an application's unit tests pass despite -changes in the libraries it uses, you can have more confidence that -the library changes haven't broken anything. Many developers -complained that the unit tests were inconvenient or took too long to -run: having the buildbot run them reduces the developer's workload to -a minimum. - -In general, having more visibility into the project is always good, -and automation makes it easier for developers to do the right thing. -When everyone can see the status of the project, developers are -encouraged to keep the tree in good working order. Unit tests that -aren't run on a regular basis tend to suffer from bitrot just like -code does: exercising them on a regular basis helps to keep them -functioning and useful. - -The current version of the Buildbot is additionally targeted at -distributed free-software projects, where resources and platforms are -only available when provided by interested volunteers. The buildslaves -are designed to require an absolute minimum of configuration, reducing -the effort a potential volunteer needs to expend to be able to -contribute a new test environment to the project. The goal is for -anyone who wishes that a given project would run on their favorite -platform should be able to offer that project a buildslave, running on -that platform, where they can verify that their portability code -works, and keeps working. - -@node System Architecture, Control Flow, History and Philosophy, Introduction -@comment node-name, next, previous, up -@section System Architecture - -The Buildbot consists of a single @code{buildmaster} and one or more -@code{buildslaves}, connected in a star topology. The buildmaster -makes all decisions about what, when, and how to build. It sends -commands to be run on the build slaves, which simply execute the -commands and return the results. (certain steps involve more local -decision making, where the overhead of sending a lot of commands back -and forth would be inappropriate, but in general the buildmaster is -responsible for everything). - -The buildmaster is usually fed @code{Changes} by some sort of version control -system (@pxref{Change Sources}), which may cause builds to be run. As the -builds are performed, various status messages are produced, which are then sent -to any registered Status Targets (@pxref{Status Delivery}). - -@c @image{FILENAME, WIDTH, HEIGHT, ALTTEXT, EXTENSION} -@image{images/overview,,,Overview Diagram,} - -The buildmaster is configured and maintained by the ``buildmaster -admin'', who is generally the project team member responsible for -build process issues. Each buildslave is maintained by a ``buildslave -admin'', who do not need to be quite as involved. Generally slaves are -run by anyone who has an interest in seeing the project work well on -their favorite platform. - -@menu -* BuildSlave Connections:: -* Buildmaster Architecture:: -* Status Delivery Architecture:: -@end menu - -@node BuildSlave Connections, Buildmaster Architecture, System Architecture, System Architecture -@subsection BuildSlave Connections - -The buildslaves are typically run on a variety of separate machines, -at least one per platform of interest. These machines connect to the -buildmaster over a TCP connection to a publically-visible port. As a -result, the buildslaves can live behind a NAT box or similar -firewalls, as long as they can get to buildmaster. The TCP connections -are initiated by the buildslave and accepted by the buildmaster, but -commands and results travel both ways within this connection. The -buildmaster is always in charge, so all commands travel exclusively -from the buildmaster to the buildslave. - -To perform builds, the buildslaves must typically obtain source code -from a CVS/SVN/etc repository. Therefore they must also be able to -reach the repository. The buildmaster provides instructions for -performing builds, but does not provide the source code itself. - -@image{images/slaves,,,BuildSlave Connections,} - -@node Buildmaster Architecture, Status Delivery Architecture, BuildSlave Connections, System Architecture -@subsection Buildmaster Architecture - -The Buildmaster consists of several pieces: - -@image{images/master,,,BuildMaster Architecture,} - -@itemize @bullet - -@item -Change Sources, which create a Change object each time something is -modified in the VC repository. Most ChangeSources listen for messages -from a hook script of some sort. Some sources actively poll the -repository on a regular basis. All Changes are fed to the Schedulers. - -@item -Schedulers, which decide when builds should be performed. They collect -Changes into BuildRequests, which are then queued for delivery to -Builders until a buildslave is available. - -@item -Builders, which control exactly @emph{how} each build is performed -(with a series of BuildSteps, configured in a BuildFactory). Each -Build is run on a single buildslave. - -@item -Status plugins, which deliver information about the build results -through protocols like HTTP, mail, and IRC. - -@end itemize - -Each Builder is configured with a list of BuildSlaves that it will use -for its builds. These buildslaves are expected to behave identically: -the only reason to use multiple BuildSlaves for a single Builder is to -provide a measure of load-balancing. - -Within a single BuildSlave, each Builder creates its own SlaveBuilder -instance. These SlaveBuilders operate independently from each other. -Each gets its own base directory to work in. It is quite common to -have many Builders sharing the same buildslave. For example, there -might be two buildslaves: one for i386, and a second for PowerPC. -There may then be a pair of Builders that do a full compile/test run, -one for each architecture, and a lone Builder that creates snapshot -source tarballs if the full builders complete successfully. The full -builders would each run on a single buildslave, whereas the tarball -creation step might run on either buildslave (since the platform -doesn't matter when creating source tarballs). In this case, the -mapping would look like: - -@example -Builder(full-i386) -> BuildSlaves(slave-i386) -Builder(full-ppc) -> BuildSlaves(slave-ppc) -Builder(source-tarball) -> BuildSlaves(slave-i386, slave-ppc) -@end example - -and each BuildSlave would have two SlaveBuilders inside it, one for a -full builder, and a second for the source-tarball builder. - -Once a SlaveBuilder is available, the Builder pulls one or more -BuildRequests off its incoming queue. (It may pull more than one if it -determines that it can merge the requests together; for example, there -may be multiple requests to build the current HEAD revision). These -requests are merged into a single Build instance, which includes the -SourceStamp that describes what exact version of the source code -should be used for the build. The Build is then randomly assigned to a -free SlaveBuilder and the build begins. - -The behaviour when BuildRequests are merged can be customized, @pxref{Merging -BuildRequests}. - -@node Status Delivery Architecture, , Buildmaster Architecture, System Architecture -@subsection Status Delivery Architecture - -The buildmaster maintains a central Status object, to which various -status plugins are connected. Through this Status object, a full -hierarchy of build status objects can be obtained. - -@image{images/status,,,Status Delivery,} - -The configuration file controls which status plugins are active. Each -status plugin gets a reference to the top-level Status object. From -there they can request information on each Builder, Build, Step, and -LogFile. This query-on-demand interface is used by the html.Waterfall -plugin to create the main status page each time a web browser hits the -main URL. - -The status plugins can also subscribe to hear about new Builds as they -occur: this is used by the MailNotifier to create new email messages -for each recently-completed Build. - -The Status object records the status of old builds on disk in the -buildmaster's base directory. This allows it to return information -about historical builds. - -There are also status objects that correspond to Schedulers and -BuildSlaves. These allow status plugins to report information about -upcoming builds, and the online/offline status of each buildslave. - - -@node Control Flow, , System Architecture, Introduction -@comment node-name, next, previous, up -@section Control Flow - -A day in the life of the buildbot: - -@itemize @bullet - -@item -A developer commits some source code changes to the repository. A hook -script or commit trigger of some sort sends information about this -change to the buildmaster through one of its configured Change -Sources. This notification might arrive via email, or over a network -connection (either initiated by the buildmaster as it ``subscribes'' -to changes, or by the commit trigger as it pushes Changes towards the -buildmaster). The Change contains information about who made the -change, what files were modified, which revision contains the change, -and any checkin comments. - -@item -The buildmaster distributes this change to all of its configured -Schedulers. Any ``important'' changes cause the ``tree-stable-timer'' -to be started, and the Change is added to a list of those that will go -into a new Build. When the timer expires, a Build is started on each -of a set of configured Builders, all compiling/testing the same source -code. Unless configured otherwise, all Builds run in parallel on the -various buildslaves. - -@item -The Build consists of a series of Steps. Each Step causes some number -of commands to be invoked on the remote buildslave associated with -that Builder. The first step is almost always to perform a checkout of -the appropriate revision from the same VC system that produced the -Change. The rest generally perform a compile and run unit tests. As -each Step runs, the buildslave reports back command output and return -status to the buildmaster. - -@item -As the Build runs, status messages like ``Build Started'', ``Step -Started'', ``Build Finished'', etc, are published to a collection of -Status Targets. One of these targets is usually the HTML ``Waterfall'' -display, which shows a chronological list of events, and summarizes -the results of the most recent build at the top of each column. -Developers can periodically check this page to see how their changes -have fared. If they see red, they know that they've made a mistake and -need to fix it. If they see green, they know that they've done their -duty and don't need to worry about their change breaking anything. - -@item -If a MailNotifier status target is active, the completion of a build -will cause email to be sent to any developers whose Changes were -incorporated into this Build. The MailNotifier can be configured to -only send mail upon failing builds, or for builds which have just -transitioned from passing to failing. Other status targets can provide -similar real-time notification via different communication channels, -like IRC. - -@end itemize - - -@node Installation, Concepts, Introduction, Top -@chapter Installation - -@menu -* Requirements:: -* Installing the code:: -* Creating a buildmaster:: -* Upgrading an Existing Buildmaster:: -* Creating a buildslave:: -* Launching the daemons:: -* Logfiles:: -* Shutdown:: -* Maintenance:: -* Troubleshooting:: -@end menu - -@node Requirements, Installing the code, Installation, Installation -@section Requirements - -At a bare minimum, you'll need the following (for both the buildmaster -and a buildslave): - -@itemize @bullet -@item -Python: http://www.python.org - -Buildbot requires python-2.3 or later, and is primarily developed -against python-2.4. It is also tested against python-2.5 . - -@item -Twisted: http://twistedmatrix.com - -Both the buildmaster and the buildslaves require Twisted-2.0.x or -later. It has been tested against all releases of Twisted up to -Twisted-2.5.0 (the most recent as of this writing). As always, the -most recent version is recommended. - -Twisted is delivered as a collection of subpackages. You'll need at -least "Twisted" (the core package), and you'll also want TwistedMail, -TwistedWeb, and TwistedWords (for sending email, serving a web status -page, and delivering build status via IRC, respectively). You might -also want TwistedConch (for the encrypted Manhole debug port). Note -that Twisted requires ZopeInterface to be installed as well. - -@end itemize - -Certain other packages may be useful on the system running the -buildmaster: - -@itemize @bullet -@item -CVSToys: http://purl.net/net/CVSToys - -If your buildmaster uses FreshCVSSource to receive change notification -from a cvstoys daemon, it will require CVSToys be installed (tested -with CVSToys-1.0.10). If the it doesn't use that source (i.e. if you -only use a mail-parsing change source, or the SVN notification -script), you will not need CVSToys. - -@end itemize - -And of course, your project's build process will impose additional -requirements on the buildslaves. These hosts must have all the tools -necessary to compile and test your project's source code. - - -@node Installing the code, Creating a buildmaster, Requirements, Installation -@section Installing the code - -@cindex installation - -The Buildbot is installed using the standard python @code{distutils} -module. After unpacking the tarball, the process is: - -@example -python setup.py build -python setup.py install -@end example - -where the install step may need to be done as root. This will put the -bulk of the code in somewhere like -/usr/lib/python2.3/site-packages/buildbot . It will also install the -@code{buildbot} command-line tool in /usr/bin/buildbot. - -To test this, shift to a different directory (like /tmp), and run: - -@example -buildbot --version -@end example - -If it shows you the versions of Buildbot and Twisted, the install went -ok. If it says @code{no such command} or it gets an @code{ImportError} -when it tries to load the libaries, then something went wrong. -@code{pydoc buildbot} is another useful diagnostic tool. - -Windows users will find these files in other places. You will need to -make sure that python can find the libraries, and will probably find -it convenient to have @code{buildbot} on your PATH. - -If you wish, you can run the buildbot unit test suite like this: - -@example -PYTHONPATH=. trial buildbot.test -@end example - -This should run up to 192 tests, depending upon what VC tools you have -installed. On my desktop machine it takes about five minutes to -complete. Nothing should fail, a few might be skipped. If any of the -tests fail, you should stop and investigate the cause before -continuing the installation process, as it will probably be easier to -track down the bug early. - -If you cannot or do not wish to install the buildbot into a site-wide -location like @file{/usr} or @file{/usr/local}, you can also install -it into the account's home directory. Do the install command like -this: - -@example -python setup.py install --home=~ -@end example - -That will populate @file{~/lib/python} and create -@file{~/bin/buildbot}. Make sure this lib directory is on your -@code{PYTHONPATH}. - - -@node Creating a buildmaster, Upgrading an Existing Buildmaster, Installing the code, Installation -@section Creating a buildmaster - -As you learned earlier (@pxref{System Architecture}), the buildmaster -runs on a central host (usually one that is publically visible, so -everybody can check on the status of the project), and controls all -aspects of the buildbot system. Let us call this host -@code{buildbot.example.org}. - -You may wish to create a separate user account for the buildmaster, -perhaps named @code{buildmaster}. This can help keep your personal -configuration distinct from that of the buildmaster and is useful if -you have to use a mail-based notification system (@pxref{Change -Sources}). However, the Buildbot will work just fine with your regular -user account. - -You need to choose a directory for the buildmaster, called the -@code{basedir}. This directory will be owned by the buildmaster, which -will use configuration files therein, and create status files as it -runs. @file{~/Buildbot} is a likely value. If you run multiple -buildmasters in the same account, or if you run both masters and -slaves, you may want a more distinctive name like -@file{~/Buildbot/master/gnomovision} or -@file{~/Buildmasters/fooproject}. If you are using a separate user -account, this might just be @file{~buildmaster/masters/fooproject}. - -Once you've picked a directory, use the @command{buildbot -create-master} command to create the directory and populate it with -startup files: - -@example -buildbot create-master @var{basedir} -@end example - -You will need to create a configuration file (@pxref{Configuration}) -before starting the buildmaster. Most of the rest of this manual is -dedicated to explaining how to do this. A sample configuration file is -placed in the working directory, named @file{master.cfg.sample}, which -can be copied to @file{master.cfg} and edited to suit your purposes. - -(Internal details: This command creates a file named -@file{buildbot.tac} that contains all the state necessary to create -the buildmaster. Twisted has a tool called @code{twistd} which can use -this .tac file to create and launch a buildmaster instance. twistd -takes care of logging and daemonization (running the program in the -background). @file{/usr/bin/buildbot} is a front end which runs twistd -for you.) - -In addition to @file{buildbot.tac}, a small @file{Makefile.sample} is -installed. This can be used as the basis for customized daemon startup, -@xref{Launching the daemons}. - -@node Upgrading an Existing Buildmaster, Creating a buildslave, Creating a buildmaster, Installation -@section Upgrading an Existing Buildmaster - -If you have just installed a new version of the Buildbot code, and you -have buildmasters that were created using an older version, you'll -need to upgrade these buildmasters before you can use them. The -upgrade process adds and modifies files in the buildmaster's base -directory to make it compatible with the new code. - -@example -buildbot upgrade-master @var{basedir} -@end example - -This command will also scan your @file{master.cfg} file for -incompatibilities (by loading it and printing any errors or deprecation -warnings that occur). Each buildbot release tries to be compatible -with configurations that worked cleanly (i.e. without deprecation -warnings) on the previous release: any functions or classes that are -to be removed will first be deprecated in a release, to give users a -chance to start using their replacement. - -The 0.7.6 release introduced the @file{public_html/} directory, which -contains @file{index.html} and other files served by the -@code{WebStatus} and @code{Waterfall} status displays. The -@code{upgrade-master} command will create these files if they do not -already exist. It will not modify existing copies, but it will write a -new copy in e.g. @file{index.html.new} if the new version differs from -the version that already exists. - -The @code{upgrade-master} command is idempotent. It is safe to run it -multiple times. After each upgrade of the buildbot code, you should -use @code{upgrade-master} on all your buildmasters. - - -@node Creating a buildslave, Launching the daemons, Upgrading an Existing Buildmaster, Installation -@section Creating a buildslave - -Typically, you will be adding a buildslave to an existing buildmaster, -to provide additional architecture coverage. The buildbot -administrator will give you several pieces of information necessary to -connect to the buildmaster. You should also be somewhat familiar with -the project being tested, so you can troubleshoot build problems -locally. - -The buildbot exists to make sure that the project's stated ``how to -build it'' process actually works. To this end, the buildslave should -run in an environment just like that of your regular developers. -Typically the project build process is documented somewhere -(@file{README}, @file{INSTALL}, etc), in a document that should -mention all library dependencies and contain a basic set of build -instructions. This document will be useful as you configure the host -and account in which the buildslave runs. - -Here's a good checklist for setting up a buildslave: - -@enumerate -@item -Set up the account - -It is recommended (although not mandatory) to set up a separate user -account for the buildslave. This account is frequently named -@code{buildbot} or @code{buildslave}. This serves to isolate your -personal working environment from that of the slave's, and helps to -minimize the security threat posed by letting possibly-unknown -contributors run arbitrary code on your system. The account should -have a minimum of fancy init scripts. - -@item -Install the buildbot code - -Follow the instructions given earlier (@pxref{Installing the code}). -If you use a separate buildslave account, and you didn't install the -buildbot code to a shared location, then you will need to install it -with @code{--home=~} for each account that needs it. - -@item -Set up the host - -Make sure the host can actually reach the buildmaster. Usually the -buildmaster is running a status webserver on the same machine, so -simply point your web browser at it and see if you can get there. -Install whatever additional packages or libraries the project's -INSTALL document advises. (or not: if your buildslave is supposed to -make sure that building without optional libraries still works, then -don't install those libraries). - -Again, these libraries don't necessarily have to be installed to a -site-wide shared location, but they must be available to your build -process. Accomplishing this is usually very specific to the build -process, so installing them to @file{/usr} or @file{/usr/local} is -usually the best approach. - -@item -Test the build process - -Follow the instructions in the INSTALL document, in the buildslave's -account. Perform a full CVS (or whatever) checkout, configure, make, -run tests, etc. Confirm that the build works without manual fussing. -If it doesn't work when you do it by hand, it will be unlikely to work -when the buildbot attempts to do it in an automated fashion. - -@item -Choose a base directory - -This should be somewhere in the buildslave's account, typically named -after the project which is being tested. The buildslave will not touch -any file outside of this directory. Something like @file{~/Buildbot} -or @file{~/Buildslaves/fooproject} is appropriate. - -@item -Get the buildmaster host/port, botname, and password - -When the buildbot admin configures the buildmaster to accept and use -your buildslave, they will provide you with the following pieces of -information: - -@itemize @bullet -@item -your buildslave's name -@item -the password assigned to your buildslave -@item -the hostname and port number of the buildmaster, i.e. buildbot.example.org:8007 -@end itemize - -@item -Create the buildslave - -Now run the 'buildbot' command as follows: - -@example -buildbot create-slave @var{BASEDIR} @var{MASTERHOST}:@var{PORT} @var{SLAVENAME} @var{PASSWORD} -@end example - -This will create the base directory and a collection of files inside, -including the @file{buildbot.tac} file that contains all the -information you passed to the @code{buildbot} command. - -@item -Fill in the hostinfo files - -When it first connects, the buildslave will send a few files up to the -buildmaster which describe the host that it is running on. These files -are presented on the web status display so that developers have more -information to reproduce any test failures that are witnessed by the -buildbot. There are sample files in the @file{info} subdirectory of -the buildbot's base directory. You should edit these to correctly -describe you and your host. - -@file{BASEDIR/info/admin} should contain your name and email address. -This is the ``buildslave admin address'', and will be visible from the -build status page (so you may wish to munge it a bit if -address-harvesting spambots are a concern). - -@file{BASEDIR/info/host} should be filled with a brief description of -the host: OS, version, memory size, CPU speed, versions of relevant -libraries installed, and finally the version of the buildbot code -which is running the buildslave. - -The optional @file{BASEDIR/info/access_uri} can specify a URI which will -connect a user to the machine. Many systems accept @code{ssh://hostname} URIs -for this purpose. - -If you run many buildslaves, you may want to create a single -@file{~buildslave/info} file and share it among all the buildslaves -with symlinks. - -@end enumerate - -@menu -* Buildslave Options:: -@end menu - -@node Buildslave Options, , Creating a buildslave, Creating a buildslave -@subsection Buildslave Options - -There are a handful of options you might want to use when creating the -buildslave with the @command{buildbot create-slave DIR } -command. You can type @command{buildbot create-slave --help} for a summary. -To use these, just include them on the @command{buildbot create-slave} -command line, like this: - -@example -buildbot create-slave --umask=022 ~/buildslave buildmaster.example.org:42012 myslavename mypasswd -@end example - -@table @code -@item --usepty -This is a boolean flag that tells the buildslave whether to launch child -processes in a PTY or with regular pipes (the default) when the master does not -specify. This option is deprecated, as this particular parameter is better -specified on the master. - -@item --umask -This is a string (generally an octal representation of an integer) -which will cause the buildslave process' ``umask'' value to be set -shortly after initialization. The ``twistd'' daemonization utility -forces the umask to 077 at startup (which means that all files created -by the buildslave or its child processes will be unreadable by any -user other than the buildslave account). If you want build products to -be readable by other accounts, you can add @code{--umask=022} to tell -the buildslave to fix the umask after twistd clobbers it. If you want -build products to be @emph{writable} by other accounts too, use -@code{--umask=000}, but this is likely to be a security problem. - -@item --keepalive -This is a number that indicates how frequently ``keepalive'' messages -should be sent from the buildslave to the buildmaster, expressed in -seconds. The default (600) causes a message to be sent to the -buildmaster at least once every 10 minutes. To set this to a lower -value, use e.g. @code{--keepalive=120}. - -If the buildslave is behind a NAT box or stateful firewall, these -messages may help to keep the connection alive: some NAT boxes tend to -forget about a connection if it has not been used in a while. When -this happens, the buildmaster will think that the buildslave has -disappeared, and builds will time out. Meanwhile the buildslave will -not realize than anything is wrong. - -@item --maxdelay -This is a number that indicates the maximum amount of time the -buildslave will wait between connection attempts, expressed in -seconds. The default (300) causes the buildslave to wait at most 5 -minutes before trying to connect to the buildmaster again. - -@item --log-size -This is the size in bytes when to rotate the Twisted log files. - -@item --log-count -This is the number of log rotations to keep around. You can either -specify a number or @code{None} (the default) to keep all -@file{twistd.log} files around. - -@end table - - -@node Launching the daemons, Logfiles, Creating a buildslave, Installation -@section Launching the daemons - -Both the buildmaster and the buildslave run as daemon programs. To -launch them, pass the working directory to the @code{buildbot} -command: - -@example -buildbot start @var{BASEDIR} -@end example - -The @var{BASEDIR} is option and can be omitted if the current directory -contains the buildbot configuration (the @file{buildbot.tac} file). - -@example -buildbot start -@end example - -This command will start the daemon and then return, so normally it -will not produce any output. To verify that the programs are indeed -running, look for a pair of files named @file{twistd.log} and -@file{twistd.pid} that should be created in the working directory. -@file{twistd.pid} contains the process ID of the newly-spawned daemon. - -When the buildslave connects to the buildmaster, new directories will -start appearing in its base directory. The buildmaster tells the slave -to create a directory for each Builder which will be using that slave. -All build operations are performed within these directories: CVS -checkouts, compiles, and tests. - -Once you get everything running, you will want to arrange for the -buildbot daemons to be started at boot time. One way is to use -@code{cron}, by putting them in a @@reboot crontab entry@footnote{this -@@reboot syntax is understood by Vixie cron, which is the flavor -usually provided with linux systems. Other unices may have a cron that -doesn't understand @@reboot}: - -@example -@@reboot buildbot start @var{BASEDIR} -@end example - -When you run @command{crontab} to set this up, remember to do it as -the buildmaster or buildslave account! If you add this to your crontab -when running as your regular account (or worse yet, root), then the -daemon will run as the wrong user, quite possibly as one with more -authority than you intended to provide. - -It is important to remember that the environment provided to cron jobs -and init scripts can be quite different that your normal runtime. -There may be fewer environment variables specified, and the PATH may -be shorter than usual. It is a good idea to test out this method of -launching the buildslave by using a cron job with a time in the near -future, with the same command, and then check @file{twistd.log} to -make sure the slave actually started correctly. Common problems here -are for @file{/usr/local} or @file{~/bin} to not be on your -@code{PATH}, or for @code{PYTHONPATH} to not be set correctly. -Sometimes @code{HOME} is messed up too. - -To modify the way the daemons are started (perhaps you want to set -some environment variables first, or perform some cleanup each time), -you can create a file named @file{Makefile.buildbot} in the base -directory. When the @file{buildbot} front-end tool is told to -@command{start} the daemon, and it sees this file (and -@file{/usr/bin/make} exists), it will do @command{make -f -Makefile.buildbot start} instead of its usual action (which involves -running @command{twistd}). When the buildmaster or buildslave is -installed, a @file{Makefile.sample} is created which implements the -same behavior as the the @file{buildbot} tool uses, so if you want to -customize the process, just copy @file{Makefile.sample} to -@file{Makefile.buildbot} and edit it as necessary. - -Some distributions may include conveniences to make starting buildbot -at boot time easy. For instance, with the default buildbot package in -Debian-based distributions, you may only need to modify -@code{/etc/default/buildbot} (see also @code{/etc/init.d/buildbot}, which -reads the configuration in @code{/etc/default/buildbot}). - -@node Logfiles, Shutdown, Launching the daemons, Installation -@section Logfiles - -@cindex logfiles - -While a buildbot daemon runs, it emits text to a logfile, named -@file{twistd.log}. A command like @code{tail -f twistd.log} is useful -to watch the command output as it runs. - -The buildmaster will announce any errors with its configuration file -in the logfile, so it is a good idea to look at the log at startup -time to check for any problems. Most buildmaster activities will cause -lines to be added to the log. - -@node Shutdown, Maintenance, Logfiles, Installation -@section Shutdown - -To stop a buildmaster or buildslave manually, use: - -@example -buildbot stop @var{BASEDIR} -@end example - -This simply looks for the @file{twistd.pid} file and kills whatever -process is identified within. - -At system shutdown, all processes are sent a @code{SIGKILL}. The -buildmaster and buildslave will respond to this by shutting down -normally. - -The buildmaster will respond to a @code{SIGHUP} by re-reading its -config file. Of course, this only works on unix-like systems with -signal support, and won't work on Windows. The following shortcut is -available: - -@example -buildbot reconfig @var{BASEDIR} -@end example - -When you update the Buildbot code to a new release, you will need to -restart the buildmaster and/or buildslave before it can take advantage -of the new code. You can do a @code{buildbot stop @var{BASEDIR}} and -@code{buildbot start @var{BASEDIR}} in quick succession, or you can -use the @code{restart} shortcut, which does both steps for you: - -@example -buildbot restart @var{BASEDIR} -@end example - -There are certain configuration changes that are not handled cleanly -by @code{buildbot reconfig}. If this occurs, @code{buildbot restart} -is a more robust tool to fully switch over to the new configuration. - -@code{buildbot restart} may also be used to start a stopped Buildbot -instance. This behaviour is useful when writing scripts that stop, start -and restart Buildbot. - -A buildslave may also be gracefully shutdown from the -@pxref{WebStatus} status plugin. This is useful to shutdown a -buildslave without interrupting any current builds. The buildmaster -will wait until the buildslave is finished all its current builds, and -will then tell the buildslave to shutdown. - -@node Maintenance, Troubleshooting, Shutdown, Installation -@section Maintenance - -It is a good idea to check the buildmaster's status page every once in -a while, to see if your buildslave is still online. Eventually the -buildbot will probably be enhanced to send you email (via the -@file{info/admin} email address) when the slave has been offline for -more than a few hours. - -If you find you can no longer provide a buildslave to the project, please -let the project admins know, so they can put out a call for a -replacement. - -The Buildbot records status and logs output continually, each time a -build is performed. The status tends to be small, but the build logs -can become quite large. Each build and log are recorded in a separate -file, arranged hierarchically under the buildmaster's base directory. -To prevent these files from growing without bound, you should -periodically delete old build logs. A simple cron job to delete -anything older than, say, two weeks should do the job. The only trick -is to leave the @file{buildbot.tac} and other support files alone, for -which find's @code{-mindepth} argument helps skip everything in the -top directory. You can use something like the following: - -@example -@@weekly cd BASEDIR && find . -mindepth 2 i-path './public_html/*' -prune -o -type f -mtime +14 -exec rm @{@} \; -@@weekly cd BASEDIR && find twistd.log* -mtime +14 -exec rm @{@} \; -@end example - -@node Troubleshooting, , Maintenance, Installation -@section Troubleshooting - -Here are a few hints on diagnosing common problems. - -@menu -* Starting the buildslave:: -* Connecting to the buildmaster:: -* Forcing Builds:: -@end menu - -@node Starting the buildslave, Connecting to the buildmaster, Troubleshooting, Troubleshooting -@subsection Starting the buildslave - -Cron jobs are typically run with a minimal shell (@file{/bin/sh}, not -@file{/bin/bash}), and tilde expansion is not always performed in such -commands. You may want to use explicit paths, because the @code{PATH} -is usually quite short and doesn't include anything set by your -shell's startup scripts (@file{.profile}, @file{.bashrc}, etc). If -you've installed buildbot (or other python libraries) to an unusual -location, you may need to add a @code{PYTHONPATH} specification (note -that python will do tilde-expansion on @code{PYTHONPATH} elements by -itself). Sometimes it is safer to fully-specify everything: - -@example -@@reboot PYTHONPATH=~/lib/python /usr/local/bin/buildbot start /usr/home/buildbot/basedir -@end example - -Take the time to get the @@reboot job set up. Otherwise, things will work -fine for a while, but the first power outage or system reboot you have will -stop the buildslave with nothing but the cries of sorrowful developers to -remind you that it has gone away. - -@node Connecting to the buildmaster, Forcing Builds, Starting the buildslave, Troubleshooting -@subsection Connecting to the buildmaster - -If the buildslave cannot connect to the buildmaster, the reason should -be described in the @file{twistd.log} logfile. Some common problems -are an incorrect master hostname or port number, or a mistyped bot -name or password. If the buildslave loses the connection to the -master, it is supposed to attempt to reconnect with an -exponentially-increasing backoff. Each attempt (and the time of the -next attempt) will be logged. If you get impatient, just manually stop -and re-start the buildslave. - -When the buildmaster is restarted, all slaves will be disconnected, -and will attempt to reconnect as usual. The reconnect time will depend -upon how long the buildmaster is offline (i.e. how far up the -exponential backoff curve the slaves have travelled). Again, -@code{buildbot stop @var{BASEDIR}; buildbot start @var{BASEDIR}} will -speed up the process. - -@node Forcing Builds, , Connecting to the buildmaster, Troubleshooting -@subsection Forcing Builds - -From the buildmaster's main status web page, you can force a build to be run on -your build slave if you set the proper authorization options in your -@code{master.cfg} file @pxref{WebStatus Configuration Parameters}. - -Figure out which column is for a builder that runs on your slave, click on that -builder's name, and the page that comes up will have a "Force Build" button. -Fill in the form, hit the button, and a moment later you should see your -slave's @file{twistd.log} filling with commands being run. Using @code{pstree} -or @code{top} should also reveal the cvs/make/gcc/etc processes being run by -the buildslave. Note that the same web page should also show the @file{admin} -and @file{host} information files that you configured earlier. - -@node Concepts, Configuration, Installation, Top -@chapter Concepts - -This chapter defines some of the basic concepts that the Buildbot -uses. You'll need to understand how the Buildbot sees the world to -configure it properly. - -@menu -* Version Control Systems:: -* Schedulers:: -* BuildSet:: -* BuildRequest:: -* Builder:: -* Users:: -* Build Properties:: -@end menu - -@node Version Control Systems, Schedulers, Concepts, Concepts -@section Version Control Systems - -@cindex Version Control - -These source trees come from a Version Control System of some kind. -CVS and Subversion are two popular ones, but the Buildbot supports -others. All VC systems have some notion of an upstream -@code{repository} which acts as a server@footnote{except Darcs, but -since the Buildbot never modifies its local source tree we can ignore -the fact that Darcs uses a less centralized model}, from which clients -can obtain source trees according to various parameters. The VC -repository provides source trees of various projects, for different -branches, and from various points in time. The first thing we have to -do is to specify which source tree we want to get. - -@menu -* Generalizing VC Systems:: -* Source Tree Specifications:: -* How Different VC Systems Specify Sources:: -* Attributes of Changes:: -@end menu - -@node Generalizing VC Systems, Source Tree Specifications, Version Control Systems, Version Control Systems -@subsection Generalizing VC Systems - -For the purposes of the Buildbot, we will try to generalize all VC -systems as having repositories that each provide sources for a variety -of projects. Each project is defined as a directory tree with source -files. The individual files may each have revisions, but we ignore -that and treat the project as a whole as having a set of revisions -(CVS is really the only VC system still in widespread use that has -per-file revisions.. everything modern has moved to atomic tree-wide -changesets). Each time someone commits a change to the project, a new -revision becomes available. These revisions can be described by a -tuple with two items: the first is a branch tag, and the second is -some kind of revision stamp or timestamp. Complex projects may have -multiple branch tags, but there is always a default branch. The -timestamp may be an actual timestamp (such as the -D option to CVS), -or it may be a monotonically-increasing transaction number (such as -the change number used by SVN and P4, or the revision number used by -Arch/Baz/Bazaar, or a labeled tag used in CVS)@footnote{many VC -systems provide more complexity than this: in particular the local -views that P4 and ClearCase can assemble out of various source -directories are more complex than we're prepared to take advantage of -here}. The SHA1 revision ID used by Monotone, Mercurial, and Git is -also a kind of revision stamp, in that it specifies a unique copy of -the source tree, as does a Darcs ``context'' file. - -When we aren't intending to make any changes to the sources we check out -(at least not any that need to be committed back upstream), there are two -basic ways to use a VC system: - -@itemize @bullet -@item -Retrieve a specific set of source revisions: some tag or key is used -to index this set, which is fixed and cannot be changed by subsequent -developers committing new changes to the tree. Releases are built from -tagged revisions like this, so that they can be rebuilt again later -(probably with controlled modifications). -@item -Retrieve the latest sources along a specific branch: some tag is used -to indicate which branch is to be used, but within that constraint we want -to get the latest revisions. -@end itemize - -Build personnel or CM staff typically use the first approach: the -build that results is (ideally) completely specified by the two -parameters given to the VC system: repository and revision tag. This -gives QA and end-users something concrete to point at when reporting -bugs. Release engineers are also reportedly fond of shipping code that -can be traced back to a concise revision tag of some sort. - -Developers are more likely to use the second approach: each morning -the developer does an update to pull in the changes committed by the -team over the last day. These builds are not easy to fully specify: it -depends upon exactly when you did a checkout, and upon what local -changes the developer has in their tree. Developers do not normally -tag each build they produce, because there is usually significant -overhead involved in creating these tags. Recreating the trees used by -one of these builds can be a challenge. Some VC systems may provide -implicit tags (like a revision number), while others may allow the use -of timestamps to mean ``the state of the tree at time X'' as opposed -to a tree-state that has been explicitly marked. - -The Buildbot is designed to help developers, so it usually works in -terms of @emph{the latest} sources as opposed to specific tagged -revisions. However, it would really prefer to build from reproducible -source trees, so implicit revisions are used whenever possible. - -@node Source Tree Specifications, How Different VC Systems Specify Sources, Generalizing VC Systems, Version Control Systems -@subsection Source Tree Specifications - -So for the Buildbot's purposes we treat each VC system as a server -which can take a list of specifications as input and produce a source -tree as output. Some of these specifications are static: they are -attributes of the builder and do not change over time. Others are more -variable: each build will have a different value. The repository is -changed over time by a sequence of Changes, each of which represents a -single developer making changes to some set of files. These Changes -are cumulative@footnote{Monotone's @emph{multiple heads} feature -violates this assumption of cumulative Changes, but in most situations -the changes don't occur frequently enough for this to be a significant -problem}. - -For normal builds, the Buildbot wants to get well-defined source trees -that contain specific Changes, and exclude other Changes that may have -occurred after the desired ones. We assume that the Changes arrive at -the buildbot (through one of the mechanisms described in @pxref{Change -Sources}) in the same order in which they are committed to the -repository. The Buildbot waits for the tree to become ``stable'' -before initiating a build, for two reasons. The first is that -developers frequently make multiple related commits in quick -succession, even when the VC system provides ways to make atomic -transactions involving multiple files at the same time. Running a -build in the middle of these sets of changes would use an inconsistent -set of source files, and is likely to fail (and is certain to be less -useful than a build which uses the full set of changes). The -tree-stable-timer is intended to avoid these useless builds that -include some of the developer's changes but not all. The second reason -is that some VC systems (i.e. CVS) do not provide repository-wide -transaction numbers, so that timestamps are the only way to refer to -a specific repository state. These timestamps may be somewhat -ambiguous, due to processing and notification delays. By waiting until -the tree has been stable for, say, 10 minutes, we can choose a -timestamp from the middle of that period to use for our source -checkout, and then be reasonably sure that any clock-skew errors will -not cause the build to be performed on an inconsistent set of source -files. - -The Schedulers always use the tree-stable-timer, with a timeout that -is configured to reflect a reasonable tradeoff between build latency -and change frequency. When the VC system provides coherent -repository-wide revision markers (such as Subversion's revision -numbers, or in fact anything other than CVS's timestamps), the -resulting Build is simply performed against a source tree defined by -that revision marker. When the VC system does not provide this, a -timestamp from the middle of the tree-stable period is used to -generate the source tree@footnote{this @code{checkoutDelay} defaults -to half the tree-stable timer, but it can be overridden with an -argument to the Source Step}. - -@node How Different VC Systems Specify Sources, Attributes of Changes, Source Tree Specifications, Version Control Systems -@subsection How Different VC Systems Specify Sources - -For CVS, the static specifications are @code{repository} and -@code{module}. In addition to those, each build uses a timestamp (or -omits the timestamp to mean @code{the latest}) and @code{branch tag} -(which defaults to HEAD). These parameters collectively specify a set -of sources from which a build may be performed. - -@uref{http://subversion.tigris.org, Subversion} combines the -repository, module, and branch into a single @code{Subversion URL} -parameter. Within that scope, source checkouts can be specified by a -numeric @code{revision number} (a repository-wide -monotonically-increasing marker, such that each transaction that -changes the repository is indexed by a different revision number), or -a revision timestamp. When branches are used, the repository and -module form a static @code{baseURL}, while each build has a -@code{revision number} and a @code{branch} (which defaults to a -statically-specified @code{defaultBranch}). The @code{baseURL} and -@code{branch} are simply concatenated together to derive the -@code{svnurl} to use for the checkout. - -@uref{http://www.perforce.com/, Perforce} is similar. The server -is specified through a @code{P4PORT} parameter. Module and branch -are specified in a single depot path, and revisions are -depot-wide. When branches are used, the @code{p4base} and -@code{defaultBranch} are concatenated together to produce the depot -path. - -@uref{http://wiki.gnuarch.org/, Arch} and -@uref{http://bazaar.canonical.com/, Bazaar} specify a repository by -URL, as well as a @code{version} which is kind of like a branch name. -Arch uses the word @code{archive} to represent the repository. Arch -lets you push changes from one archive to another, removing the strict -centralization required by CVS and SVN. It retains the distinction -between repository and working directory that most other VC systems -use. For complex multi-module directory structures, Arch has a -built-in @code{build config} layer with which the checkout process has -two steps. First, an initial bootstrap checkout is performed to -retrieve a set of build-config files. Second, one of these files is -used to figure out which archives/modules should be used to populate -subdirectories of the initial checkout. - -Builders which use Arch and Bazaar therefore have a static archive -@code{url}, and a default ``branch'' (which is a string that specifies -a complete category--branch--version triple). Each build can have its -own branch (the category--branch--version string) to override the -default, as well as a revision number (which is turned into a ---patch-NN suffix when performing the checkout). - - -@uref{http://bazaar-vcs.org, Bzr} (which is a descendant of -Arch/Bazaar, and is frequently referred to as ``Bazaar'') has the same -sort of repository-vs-workspace model as Arch, but the repository data -can either be stored inside the working directory or kept elsewhere -(either on the same machine or on an entirely different machine). For -the purposes of Buildbot (which never commits changes), the repository -is specified with a URL and a revision number. - -The most common way to obtain read-only access to a bzr tree is via -HTTP, simply by making the repository visible through a web server -like Apache. Bzr can also use FTP and SFTP servers, if the buildslave -process has sufficient privileges to access them. Higher performance -can be obtained by running a special Bazaar-specific server. None of -these matter to the buildbot: the repository URL just has to match the -kind of server being used. The @code{repoURL} argument provides the -location of the repository. - -Branches are expressed as subdirectories of the main central -repository, which means that if branches are being used, the BZR step -is given a @code{baseURL} and @code{defaultBranch} instead of getting -the @code{repoURL} argument. - - -@uref{http://darcs.net/, Darcs} doesn't really have the -notion of a single master repository. Nor does it really have -branches. In Darcs, each working directory is also a repository, and -there are operations to push and pull patches from one of these -@code{repositories} to another. For the Buildbot's purposes, all you -need to do is specify the URL of a repository that you want to build -from. The build slave will then pull the latest patches from that -repository and build them. Multiple branches are implemented by using -multiple repositories (possibly living on the same server). - -Builders which use Darcs therefore have a static @code{repourl} which -specifies the location of the repository. If branches are being used, -the source Step is instead configured with a @code{baseURL} and a -@code{defaultBranch}, and the two strings are simply concatenated -together to obtain the repository's URL. Each build then has a -specific branch which replaces @code{defaultBranch}, or just uses the -default one. Instead of a revision number, each build can have a -``context'', which is a string that records all the patches that are -present in a given tree (this is the output of @command{darcs changes ---context}, and is considerably less concise than, e.g. Subversion's -revision number, but the patch-reordering flexibility of Darcs makes -it impossible to provide a shorter useful specification). - -@uref{http://selenic.com/mercurial, Mercurial} is like Darcs, in that -each branch is stored in a separate repository. The @code{repourl}, -@code{baseURL}, and @code{defaultBranch} arguments are all handled the -same way as with Darcs. The ``revision'', however, is the hash -identifier returned by @command{hg identify}. - -@uref{http://git.or.cz/, Git} also follows a decentralized model, and -each repository can have several branches and tags. The source Step is -configured with a static @code{repourl} which specifies the location -of the repository. In addition, an optional @code{branch} parameter -can be specified to check out code from a specific branch instead of -the default ``master'' branch. The ``revision'' is specified as a SHA1 -hash as returned by e.g. @command{git rev-parse}. No attempt is made -to ensure that the specified revision is actually a subset of the -specified branch. - - -@node Attributes of Changes, , How Different VC Systems Specify Sources, Version Control Systems -@subsection Attributes of Changes - -@heading Who - -Each Change has a @code{who} attribute, which specifies which -developer is responsible for the change. This is a string which comes -from a namespace controlled by the VC repository. Frequently this -means it is a username on the host which runs the repository, but not -all VC systems require this (Arch, for example, uses a fully-qualified -@code{Arch ID}, which looks like an email address, as does Darcs). -Each StatusNotifier will map the @code{who} attribute into something -appropriate for their particular means of communication: an email -address, an IRC handle, etc. - -@heading Files - -It also has a list of @code{files}, which are just the tree-relative -filenames of any files that were added, deleted, or modified for this -Change. These filenames are used by the @code{fileIsImportant} -function (in the Scheduler) to decide whether it is worth triggering a -new build or not, e.g. the function could use the following function -to only run a build if a C file were checked in: - -@example -def has_C_files(change): - for name in change.files: - if name.endswith(".c"): - return True - return False -@end example - -Certain BuildSteps can also use the list of changed files -to run a more targeted series of tests, e.g. the -@code{python_twisted.Trial} step can run just the unit tests that -provide coverage for the modified .py files instead of running the -full test suite. - -@heading Comments - -The Change also has a @code{comments} attribute, which is a string -containing any checkin comments. - -@heading Revision - -Each Change can have a @code{revision} attribute, which describes how -to get a tree with a specific state: a tree which includes this Change -(and all that came before it) but none that come after it. If this -information is unavailable, the @code{.revision} attribute will be -@code{None}. These revisions are provided by the ChangeSource, and -consumed by the @code{computeSourceRevision} method in the appropriate -@code{source.Source} class. - -@table @samp -@item CVS -@code{revision} is an int, seconds since the epoch -@item SVN -@code{revision} is an int, the changeset number (r%d) -@item Darcs -@code{revision} is a large string, the output of @code{darcs changes --context} -@item Mercurial -@code{revision} is a short string (a hash ID), the output of @code{hg identify} -@item Arch/Bazaar -@code{revision} is the full revision ID (ending in --patch-%d) -@item P4 -@code{revision} is an int, the transaction number -@item Git -@code{revision} is a short string (a SHA1 hash), the output of e.g. -@code{git rev-parse} -@end table - -@heading Branches - -The Change might also have a @code{branch} attribute. This indicates -that all of the Change's files are in the same named branch. The -Schedulers get to decide whether the branch should be built or not. - -For VC systems like CVS, Arch, Monotone, and Git, the @code{branch} -name is unrelated to the filename. (that is, the branch name and the -filename inhabit unrelated namespaces). For SVN, branches are -expressed as subdirectories of the repository, so the file's -``svnurl'' is a combination of some base URL, the branch name, and the -filename within the branch. (In a sense, the branch name and the -filename inhabit the same namespace). Darcs branches are -subdirectories of a base URL just like SVN. Mercurial branches are the -same as Darcs. - -@table @samp -@item CVS -branch='warner-newfeature', files=['src/foo.c'] -@item SVN -branch='branches/warner-newfeature', files=['src/foo.c'] -@item Darcs -branch='warner-newfeature', files=['src/foo.c'] -@item Mercurial -branch='warner-newfeature', files=['src/foo.c'] -@item Arch/Bazaar -branch='buildbot--usebranches--0', files=['buildbot/master.py'] -@item Git -branch='warner-newfeature', files=['src/foo.c'] -@end table - -@heading Build Properties - -A Change may have one or more properties attached to it, usually specified -through the Force Build form or @pxref{sendchange}. Properties are discussed -in detail in the @pxref{Build Properties} section. - -@heading Links - -@c TODO: who is using 'links'? how is it being used? - -Finally, the Change might have a @code{links} list, which is intended -to provide a list of URLs to a @emph{viewcvs}-style web page that -provides more detail for this Change, perhaps including the full file -diffs. - - -@node Schedulers, BuildSet, Version Control Systems, Concepts -@section Schedulers - -@cindex Scheduler - -Each Buildmaster has a set of @code{Scheduler} objects, each of which -gets a copy of every incoming Change. The Schedulers are responsible -for deciding when Builds should be run. Some Buildbot installations -might have a single Scheduler, while others may have several, each for -a different purpose. - -For example, a ``quick'' scheduler might exist to give immediate -feedback to developers, hoping to catch obvious problems in the code -that can be detected quickly. These typically do not run the full test -suite, nor do they run on a wide variety of platforms. They also -usually do a VC update rather than performing a brand-new checkout -each time. You could have a ``quick'' scheduler which used a 30 second -timeout, and feeds a single ``quick'' Builder that uses a VC -@code{mode='update'} setting. - -A separate ``full'' scheduler would run more comprehensive tests a -little while later, to catch more subtle problems. This scheduler -would have a longer tree-stable-timer, maybe 30 minutes, and would -feed multiple Builders (with a @code{mode=} of @code{'copy'}, -@code{'clobber'}, or @code{'export'}). - -The @code{tree-stable-timer} and @code{fileIsImportant} decisions are -made by the Scheduler. Dependencies are also implemented here. -Periodic builds (those which are run every N seconds rather than after -new Changes arrive) are triggered by a special @code{Periodic} -Scheduler subclass. The default Scheduler class can also be told to -watch for specific branches, ignoring Changes on other branches. This -may be useful if you have a trunk and a few release branches which -should be tracked, but when you don't want to have the Buildbot pay -attention to several dozen private user branches. - -When the setup has multiple sources of Changes the @code{category} -can be used for @code{Scheduler} objects to filter out a subset -of the Changes. Note that not all change sources can attach a category. - -Some Schedulers may trigger builds for other reasons, other than -recent Changes. For example, a Scheduler subclass could connect to a -remote buildmaster and watch for builds of a library to succeed before -triggering a local build that uses that library. - -Each Scheduler creates and submits @code{BuildSet} objects to the -@code{BuildMaster}, which is then responsible for making sure the -individual @code{BuildRequests} are delivered to the target -@code{Builders}. - -@code{Scheduler} instances are activated by placing them in the -@code{c['schedulers']} list in the buildmaster config file. Each -Scheduler has a unique name. - - -@node BuildSet, BuildRequest, Schedulers, Concepts -@section BuildSet - -@cindex BuildSet - -A @code{BuildSet} is the name given to a set of Builds that all -compile/test the same version of the tree on multiple Builders. In -general, all these component Builds will perform the same sequence of -Steps, using the same source code, but on different platforms or -against a different set of libraries. - -The @code{BuildSet} is tracked as a single unit, which fails if any of -the component Builds have failed, and therefore can succeed only if -@emph{all} of the component Builds have succeeded. There are two kinds -of status notification messages that can be emitted for a BuildSet: -the @code{firstFailure} type (which fires as soon as we know the -BuildSet will fail), and the @code{Finished} type (which fires once -the BuildSet has completely finished, regardless of whether the -overall set passed or failed). - -A @code{BuildSet} is created with a @emph{source stamp} tuple of -(branch, revision, changes, patch), some of which may be None, and a -list of Builders on which it is to be run. They are then given to the -BuildMaster, which is responsible for creating a separate -@code{BuildRequest} for each Builder. - -There are a couple of different likely values for the -@code{SourceStamp}: - -@table @code -@item (revision=None, changes=[CHANGES], patch=None) -This is a @code{SourceStamp} used when a series of Changes have -triggered a build. The VC step will attempt to check out a tree that -contains CHANGES (and any changes that occurred before CHANGES, but -not any that occurred after them). - -@item (revision=None, changes=None, patch=None) -This builds the most recent code on the default branch. This is the -sort of @code{SourceStamp} that would be used on a Build that was -triggered by a user request, or a Periodic scheduler. It is also -possible to configure the VC Source Step to always check out the -latest sources rather than paying attention to the Changes in the -SourceStamp, which will result in same behavior as this. - -@item (branch=BRANCH, revision=None, changes=None, patch=None) -This builds the most recent code on the given BRANCH. Again, this is -generally triggered by a user request or Periodic build. - -@item (revision=REV, changes=None, patch=(LEVEL, DIFF, SUBDIR_ROOT)) -This checks out the tree at the given revision REV, then applies a -patch (using @code{patch -pLEVEL 'Account -Activity' and verifying EC2 is listed. -@end itemize - -@node Create an AMI, Configure the Master with an EC2LatentBuildSlave, Get an AWS EC2 Account, Amazon Web Services Elastic Compute Cloud ("AWS EC2") -@subsubsection Create an AMI - -Now you need to create an AMI and configure the master. You may need to -run through this cycle a few times to get it working, but these instructions -should get you started. - -Creating an AMI is out of the scope of this document. The -@url{http://docs.amazonwebservices.com/AWSEC2/latest/GettingStartedGuide/,,EC2 Getting Started Guide} -is a good resource for this task. Here are a few additional hints. - -@itemize @bullet -@item -When an instance of the image starts, it needs to automatically start a -buildbot slave that connects to your master (to create a buildbot slave, -@pxref{Creating a buildslave}; to make a daemon, -@pxref{Launching the daemons}). - -@item -You may want to make an instance of the buildbot slave, configure it as a -standard buildslave in the master (i.e., not as a latent slave), and test and -debug it that way before you turn it into an AMI and convert to a latent -slave in the master. -@end itemize - -@node Configure the Master with an EC2LatentBuildSlave, , Create an AMI, Amazon Web Services Elastic Compute Cloud ("AWS EC2") -@subsubsection Configure the Master with an EC2LatentBuildSlave - -Now let's assume you have an AMI that should work with the -EC2LatentBuildSlave. It's now time to set up your buildbot master -configuration. - -You will need some information from your AWS account: the "Access Key Id" and -the "Secret Access Key". If you've built the AMI yourself, you probably -already are familiar with these values. If you have not, and someone has -given you access to an AMI, these hints may help you find the necessary -values: - -@itemize @bullet -@item -While logged into your AWS account, find the "Access Identifiers" link (either -on the left, or via "Your Account" -> "Access Identifiers". - -@item -On the page, you'll see alphanumeric values for "Your Access Key Id:" and -"Your Secret Access Key:". Make a note of these. Later on, we'll call the -first one your "identifier" and the second one your "secret_identifier." -@end itemize - -When creating an EC2LatentBuildSlave in the buildbot master configuration, -the first three arguments are required. The name and password are the first -two arguments, and work the same as with normal buildslaves. The next -argument specifies the type of the EC2 virtual machine (available options as -of this writing include "m1.small", "m1.large", 'm1.xlarge", "c1.medium", -and "c1.xlarge"; see the EC2 documentation for descriptions of these -machines). - -Here is the simplest example of configuring an EC2 latent buildslave. It -specifies all necessary remaining values explicitly in the instantiation. - -@example -from buildbot.ec2buildslave import EC2LatentBuildSlave -c['slaves'] = [EC2LatentBuildSlave('bot1', 'sekrit', 'm1.large', - ami='ami-12345', - identifier='publickey', - secret_identifier='privatekey' - )] -@end example - -The "ami" argument specifies the AMI that the master should start. The -"identifier" argument specifies the AWS "Access Key Id," and the -"secret_identifier" specifies the AWS "Secret Access Key." Both the AMI and -the account information can be specified in alternate ways. - -Note that whoever has your identifier and secret_identifier values can request -AWS work charged to your account, so these values need to be carefully -protected. Another way to specify these access keys is to put them in a -separate file. You can then make the access privileges stricter for this -separate file, and potentially let more people read your main configuration -file. - -By default, you can make an .ec2 directory in the home folder of the user -running the buildbot master. In that directory, create a file called aws_id. -The first line of that file should be your access key id; the second line -should be your secret access key id. Then you can instantiate the build slave -as follows. - -@example -from buildbot.ec2buildslave import EC2LatentBuildSlave -c['slaves'] = [EC2LatentBuildSlave('bot1', 'sekrit', 'm1.large', - ami='ami-12345')] -@end example - -If you want to put the key information in another file, use the -"aws_id_file_path" initialization argument. - -Previous examples used a particular AMI. If the Buildbot master will be -deployed in a process-controlled environment, it may be convenient to -specify the AMI more flexibly. Rather than specifying an individual AMI, -specify one or two AMI filters. - -In all cases, the AMI that sorts last by its location (the S3 bucket and -manifest name) will be preferred. - -One available filter is to specify the acceptable AMI owners, by AWS account -number (the 12 digit number, usually rendered in AWS with hyphens like -"1234-5678-9012", should be entered as in integer). - -@example -from buildbot.ec2buildslave import EC2LatentBuildSlave -bot1 = EC2LatentBuildSlave('bot1', 'sekrit', 'm1.large', - valid_ami_owners=[11111111111, - 22222222222], - identifier='publickey', - secret_identifier='privatekey' - ) -@end example - -The other available filter is to provide a regular expression string that -will be matched against each AMI's location (the S3 bucket and manifest name). - -@example -from buildbot.ec2buildslave import EC2LatentBuildSlave -bot1 = EC2LatentBuildSlave( - 'bot1', 'sekrit', 'm1.large', - valid_ami_location_regex=r'buildbot\-.*/image.manifest.xml', - identifier='publickey', secret_identifier='privatekey') -@end example - -The regular expression can specify a group, which will be preferred for the -sorting. Only the first group is used; subsequent groups are ignored. - -@example -from buildbot.ec2buildslave import EC2LatentBuildSlave -bot1 = EC2LatentBuildSlave( - 'bot1', 'sekrit', 'm1.large', - valid_ami_location_regex=r'buildbot\-.*\-(.*)/image.manifest.xml', - identifier='publickey', secret_identifier='privatekey') -@end example - -If the group can be cast to an integer, it will be. This allows 10 to sort -after 1, for instance. - -@example -from buildbot.ec2buildslave import EC2LatentBuildSlave -bot1 = EC2LatentBuildSlave( - 'bot1', 'sekrit', 'm1.large', - valid_ami_location_regex=r'buildbot\-.*\-(\d+)/image.manifest.xml', - identifier='publickey', secret_identifier='privatekey') -@end example - -In addition to using the password as a handshake between the master and the -slave, you may want to use a firewall to assert that only machines from a -specific IP can connect as slaves. This is possible with AWS EC2 by using -the Elastic IP feature. To configure, generate a Elastic IP in AWS, and then -specify it in your configuration using the "elastic_ip" argument. - -@example -from buildbot.ec2buildslave import EC2LatentBuildSlave -c['slaves'] = [EC2LatentBuildSlave('bot1', 'sekrit', 'm1.large', - 'ami-12345', - identifier='publickey', - secret_identifier='privatekey', - elastic_ip='208.77.188.166' - )] -@end example - -The EC2LatentBuildSlave supports all other configuration from the standard -BuildSlave. The "missing_timeout" and "notify_on_missing" specify how long -to wait for an EC2 instance to attach before considering the attempt to have -failed, and email addresses to alert, respectively. "missing_timeout" -defaults to 20 minutes. - -The "build_wait_timeout" allows you to specify how long an EC2LatentBuildSlave -should wait after a build for another build before it shuts down the EC2 -instance. It defaults to 10 minutes. - -"keypair_name" and "security_name" allow you to specify different names for -these AWS EC2 values. They both default to "latent_buildbot_slave". - -@node Dangers with Latent Buildslaves, Writing New Latent Buildslaves, Amazon Web Services Elastic Compute Cloud ("AWS EC2"), On-Demand ("Latent") Buildslaves -@subsection Dangers with Latent Buildslaves - -Any latent build slave that interacts with a for-fee service, such as the -EC2LatentBuildSlave, brings significant risks. As already identified, the -configuraton will need access to account information that, if obtained by a -criminal, can be used to charge services to your account. Also, bugs in the -buildbot software may lead to unnecessary charges. In particular, if the -master neglects to shut down an instance for some reason, a virtual machine -may be running unnecessarily, charging against your account. Manual and/or -automatic (e.g. nagios with a plugin using a library like boto) -double-checking may be appropriate. - -A comparitively trivial note is that currently if two instances try to attach -to the same latent buildslave, it is likely that the system will become -confused. This should not occur, unless, for instance, you configure a normal -build slave to connect with the authentication of a latent buildbot. If the -situation occurs, stop all attached instances and restart the master. - -@node Writing New Latent Buildslaves, , Dangers with Latent Buildslaves, On-Demand ("Latent") Buildslaves -@subsection Writing New Latent Buildslaves - -Writing a new latent buildslave should only require subclassing -@code{buildbot.buildslave.AbstractLatentBuildSlave} and implementing -start_instance and stop_instance. - -@example -def start_instance(self): - # responsible for starting instance that will try to connect with this - # master. Should return deferred. Problems should use an errback. The - # callback value can be None, or can be an iterable of short strings to - # include in the "substantiate success" status message, such as - # identifying the instance that started. - raise NotImplementedError - -def stop_instance(self, fast=False): - # responsible for shutting down instance. Return a deferred. If `fast`, - # we're trying to shut the master down, so callback as soon as is safe. - # Callback value is ignored. - raise NotImplementedError -@end example - -See @code{buildbot.ec2buildslave.EC2LatentBuildSlave} for an example, or see the -test example @code{buildbot.test_slaves.FakeLatentBuildSlave}. - -@node Defining Global Properties, Defining Builders, On-Demand ("Latent") Buildslaves, Configuration -@section Defining Global Properties -@bcindex c['properties'] -@cindex Properties - -The @code{'properties'} configuration key defines a dictionary -of properties that will be available to all builds started by the -buildmaster: - -@example -c['properties'] = @{ - 'Widget-version' : '1.2', - 'release-stage' : 'alpha' -@} -@end example - -@node Defining Builders -@section Defining Builders - -@bcindex c['builders'] - -The @code{c['builders']} key is a list of objects giving configuration for the -Builders. For more information, @xref{Builder}. The class definition for the -builder configuration is in @code{buildbot.config}. In the configuration file, -its use looks like: - -@example -from buildbot.config import BuilderConfig -c['builders'] = [ - BuilderConfig(name='quick', slavenames=['bot1', 'bot2'], factory=f_quick), - BuilderConfig(name='thorough', slavename='bot1', factory=f_thorough), -] -@end example - -The constructor takes the following keyword arguments: - -@table @code -@item name -This specifies the Builder's name, which is used in status -reports. - -@item slavename -@item slavenames -These arguments specify the buildslave or buildslaves that will be used by this -Builder. All slaves names must appear in the @code{c['slaves']} list. Each -buildslave can accomodate multiple Builders. The @code{slavenames} parameter -can be a list of names, while @code{slavename} can specify only one slave. - -@item factory -This is a @code{buildbot.process.factory.BuildFactory} instance which -controls how the build is performed. Full details appear in their own -section, @xref{Build Process}. Parameters like the location of the CVS -repository and the compile-time options used for the build are -generally provided as arguments to the factory's constructor. - -@end table - -Other optional keys may be set on each Builder: - -@table @code - -@item builddir -Specifies the name of a subdirectory (under the base directory) in which -everything related to this builder will be placed on the buildmaster. -This holds build status information. If not set, defaults to @code{name} -with some characters escaped. Each builder must have a unique build -directory. - -@item slavebuilddir -Specifies the name of a subdirectory (under the base directory) in which -everything related to this builder will be placed on the buildslave. -This is where checkouts, compiles, and tests are run. If not set, -defaults to @code{builddir}. If a slave is connected to multiple builders -that share the same @code{slavebuilddir}, make sure the slave is set to -run one build at a time or ensure this is fine to run multiple builds from -the same directory simultaneously. - -@item category -If provided, this is a string that identifies a category for the -builder to be a part of. Status clients can limit themselves to a -subset of the available categories. A common use for this is to add -new builders to your setup (for a new module, or for a new buildslave) -that do not work correctly yet and allow you to integrate them with -the active builders. You can put these new builders in a test -category, make your main status clients ignore them, and have only -private status clients pick them up. As soon as they work, you can -move them over to the active category. - -@item nextSlave -If provided, this is a function that controls which slave will be assigned -future jobs. The function is passed two arguments, the @code{Builder} -object which is assigning a new job, and a list of @code{BuildSlave} -objects. The function should return one of the @code{BuildSlave} -objects, or @code{None} if none of the available slaves should be -used. - -@item nextBuild -If provided, this is a function that controls which build request will be -handled next. The function is passed two arguments, the @code{Builder} -object which is assigning a new job, and a list of @code{BuildRequest} -objects of pending builds. The function should return one of the -@code{BuildRequest} objects, or @code{None} if none of the pending -builds should be started. - -@item locks -This argument specifies a list of locks that apply to this builder; @xref{Interlocks}. - -@item env -A Builder may be given a dictionary of environment variables in this parameter. -The variables are used in @pxref{ShellCommand} steps in builds created by this -builder. The environment variables will override anything in the buildslave's -environment. Variables passed directly to a @code{ShellCommand} will override -variables of the same name passed to the Builder. - -For example, if you a pool of identical slaves it is often easier to manage -variables like PATH from Buildbot rather than manually editing it inside of -the slaves' environment. - -@example -f = factory.BuildFactory -f.addStep(ShellCommand( - command=['bash', './configure'])) -f.addStep(Compile()) - -c['builders'] = [ - BuilderConfig(name='test', factory=f, - slavenames=['slave1', 'slave2', 'slave3', 'slave4'], - env=@{'PATH': '/opt/local/bin:/opt/app/bin:/usr/local/bin:/usr/bin'@}), -] -@end example - -@end table - - -@node Defining Status Targets -@section Defining Status Targets - -The Buildmaster has a variety of ways to present build status to -various users. Each such delivery method is a ``Status Target'' object -in the configuration's @code{status} list. To add status targets, you -just append more objects to this list: - -@bcindex c['status'] - -@example -c['status'] = [] - -from buildbot.status import html -c['status'].append(html.Waterfall(http_port=8010)) - -from buildbot.status import mail -m = mail.MailNotifier(fromaddr="buildbot@@localhost", - extraRecipients=["builds@@lists.example.com"], - sendToInterestedUsers=False) -c['status'].append(m) - -from buildbot.status import words -c['status'].append(words.IRC(host="irc.example.com", nick="bb", - channels=["#example"])) -@end example - -Status delivery has its own chapter, @xref{Status Delivery}, in which -all the built-in status targets are documented. - -@node Limiting Memory and Disk Usage -@section Limiting Memory and Disk Usage - -Buildbot stores historical information on disk in the form of "Pickle" files -and compressed logfiles. In a large installation, these can quickly consume -disk space, yet in many cases developers never consult this historical -information. Four configuration parameters control the "pruning" of various -data, shown here with their default values: - -@bcindex c['buildHorizon'] -@bcindex c['eventHorizon'] -@bcindex c['logHorizon'] -@bcindex c['buildCacheSize'] -@example -c['buildHorizon'] = 100 -c['eventHorizon'] = 50 -c['logHorizon'] = 40 - -c['buildCacheSize'] = 15 - -c['builders'] = [ - @{'name': 'test1', 'builddir': 'test1', 'factory': f, 'buildHorizon': 150@} -] -@end example - -The @code{buildHorizon} specifies the minimum number of builds for each builder -which should be kept on disk. The @code{eventHorizon} specifies the minumum -number of events to keep -- events mostly describe connections and -disconnections of slaves, and are seldom helpful to developers. The -@code{logHorizon} gives the minimum number of builds for which logs should be -maintained; this parameter must be less than @code{buildHorizon}. Builds older -than @code{logHorizon} but not older than @code{buildHorizon} will maintain -their overall status and the status of each step, but the logfiles will be -deleted. - -Finally, the @code{buildCacheSize} gives the number of builds for each builder -which are cached in memory. This number should be larger than the number of -builds required for commonly-used status displays (the waterfall or grid -views), so that those displays do not miss the cache on a refresh. - -@node Debug options -@section Debug options - - -@bcindex c['debugPassword'] -If you set @code{c['debugPassword']}, then you can connect to the -buildmaster with the diagnostic tool launched by @code{buildbot -debugclient MASTER:PORT}. From this tool, you can reload the config -file, manually force builds, and inject changes, which may be useful -for testing your buildmaster without actually commiting changes to -your repository (or before you have the Change Sources set up). The -debug tool uses the same port number as the slaves do: -@code{c['slavePortnum']}, and is authenticated with this password. - -@example -c['debugPassword'] = "debugpassword" -@end example - -@bcindex c['manhole'] -If you set @code{c['manhole']} to an instance of one of the classes in -@code{buildbot.manhole}, you can telnet or ssh into the buildmaster -and get an interactive Python shell, which may be useful for debugging -buildbot internals. It is probably only useful for buildbot -developers. It exposes full access to the buildmaster's account -(including the ability to modify and delete files), so it should not -be enabled with a weak or easily guessable password. - -There are three separate @code{Manhole} classes. Two of them use SSH, -one uses unencrypted telnet. Two of them use a username+password -combination to grant access, one of them uses an SSH-style -@file{authorized_keys} file which contains a list of ssh public keys. - -@table @code -@item manhole.AuthorizedKeysManhole -You construct this with the name of a file that contains one SSH -public key per line, just like @file{~/.ssh/authorized_keys}. If you -provide a non-absolute filename, it will be interpreted relative to -the buildmaster's base directory. - -@item manhole.PasswordManhole -This one accepts SSH connections but asks for a username and password -when authenticating. It accepts only one such pair. - - -@item manhole.TelnetManhole -This accepts regular unencrypted telnet connections, and asks for a -username/password pair before providing access. Because this -username/password is transmitted in the clear, and because Manhole -access to the buildmaster is equivalent to granting full shell -privileges to both the buildmaster and all the buildslaves (and to all -accounts which then run code produced by the buildslaves), it is -highly recommended that you use one of the SSH manholes instead. - -@end table - -@example -# some examples: -from buildbot import manhole -c['manhole'] = manhole.AuthorizedKeysManhole(1234, "authorized_keys") -c['manhole'] = manhole.PasswordManhole(1234, "alice", "mysecretpassword") -c['manhole'] = manhole.TelnetManhole(1234, "bob", "snoop_my_password_please") -@end example - -The @code{Manhole} instance can be configured to listen on a specific -port. You may wish to have this listening port bind to the loopback -interface (sometimes known as ``lo0'', ``localhost'', or 127.0.0.1) to -restrict access to clients which are running on the same host. - -@example -from buildbot.manhole import PasswordManhole -c['manhole'] = PasswordManhole("tcp:9999:interface=127.0.0.1","admin","passwd") -@end example - -To have the @code{Manhole} listen on all interfaces, use -@code{"tcp:9999"} or simply 9999. This port specification uses -@code{twisted.application.strports}, so you can make it listen on SSL -or even UNIX-domain sockets if you want. - -Note that using any Manhole requires that the TwistedConch package be -installed, and that you be using Twisted version 2.0 or later. - -The buildmaster's SSH server will use a different host key than the -normal sshd running on a typical unix host. This will cause the ssh -client to complain about a ``host key mismatch'', because it does not -realize there are two separate servers running on the same host. To -avoid this, use a clause like the following in your @file{.ssh/config} -file: - -@example -Host remotehost-buildbot - HostName remotehost - HostKeyAlias remotehost-buildbot - Port 9999 - # use 'user' if you use PasswordManhole and your name is not 'admin'. - # if you use AuthorizedKeysManhole, this probably doesn't matter. - User admin -@end example - - -@node Getting Source Code Changes, Build Process, Configuration, Top -@chapter Getting Source Code Changes - -The most common way to use the Buildbot is centered around the idea of -@code{Source Trees}: a directory tree filled with source code of some form -which can be compiled and/or tested. Some projects use languages that don't -involve any compilation step: nevertheless there may be a @code{build} phase -where files are copied or rearranged into a form that is suitable for -installation. Some projects do not have unit tests, and the Buildbot is -merely helping to make sure that the sources can compile correctly. But in -all of these cases, the thing-being-tested is a single source tree. - -A Version Control System mantains a source tree, and tells the -buildmaster when it changes. The first step of each Build is typically -to acquire a copy of some version of this tree. - -This chapter describes how the Buildbot learns about what Changes have -occurred. For more information on VC systems and Changes, see -@ref{Version Control Systems}. - - -@menu -* Change Sources:: -* Choosing ChangeSources:: -* CVSToys - PBService:: -* Mail-parsing ChangeSources:: -* PBChangeSource:: -* P4Source:: -* BonsaiPoller:: -* SVNPoller:: -* MercurialHook:: -* Bzr Hook:: -* Bzr Poller:: -@end menu - - - -@node Change Sources, Choosing ChangeSources, Getting Source Code Changes, Getting Source Code Changes -@section Change Sources - -@c TODO: rework this, the one-buildmaster-one-tree thing isn't quite -@c so narrow-minded anymore - -Each Buildmaster watches a single source tree. Changes can be provided -by a variety of ChangeSource types, however any given project will -typically have only a single ChangeSource active. This section -provides a description of all available ChangeSource types and -explains how to set up each of them. - -There are a variety of ChangeSources available, some of which are -meant to be used in conjunction with other tools to deliver Change -events from the VC repository to the buildmaster. - -@itemize @bullet - -@item CVSToys -This ChangeSource opens a TCP connection from the buildmaster to a -waiting FreshCVS daemon that lives on the repository machine, and -subscribes to hear about Changes. - -@item MaildirSource -This one watches a local maildir-format inbox for email sent out by -the repository when a change is made. When a message arrives, it is -parsed to create the Change object. A variety of parsing functions are -available to accomodate different email-sending tools. - -@item PBChangeSource -This ChangeSource listens on a local TCP socket for inbound -connections from a separate tool. Usually, this tool would be run on -the VC repository machine in a commit hook. It is expected to connect -to the TCP socket and send a Change message over the network -connection. The @command{buildbot sendchange} command is one example -of a tool that knows how to send these messages, so you can write a -commit script for your VC system that calls it to deliver the Change. -There are other tools in the contrib/ directory that use the same -protocol. - -@end itemize - -As a quick guide, here is a list of VC systems and the ChangeSources -that might be useful with them. All of these ChangeSources are in the -@code{buildbot.changes} module. - -@table @code -@item CVS - -@itemize @bullet -@item freshcvs.FreshCVSSource (connected via TCP to the freshcvs daemon) -@item mail.FCMaildirSource (watching for email sent by a freshcvs daemon) -@item mail.BonsaiMaildirSource (watching for email sent by Bonsai) -@item mail.SyncmailMaildirSource (watching for email sent by syncmail) -@item pb.PBChangeSource (listening for connections from @code{buildbot -sendchange} run in a loginfo script) -@item pb.PBChangeSource (listening for connections from a long-running -@code{contrib/viewcvspoll.py} polling process which examines the ViewCVS -database directly -@end itemize - -@item SVN -@itemize @bullet -@item pb.PBChangeSource (listening for connections from -@code{contrib/svn_buildbot.py} run in a postcommit script) -@item pb.PBChangeSource (listening for connections from a long-running -@code{contrib/svn_watcher.py} or @code{contrib/svnpoller.py} polling -process -@item mail.SVNCommitEmailMaildirSource (watching for email sent by commit-email.pl) -@item svnpoller.SVNPoller (polling the SVN repository) -@end itemize - -@item Darcs -@itemize @bullet -@item pb.PBChangeSource (listening for connections from -@code{contrib/darcs_buildbot.py} in a commit script -@end itemize - -@item Mercurial -@itemize @bullet -@item pb.PBChangeSource (listening for connections from -@code{contrib/hg_buildbot.py} run in an 'incoming' hook) -@item pb.PBChangeSource (listening for connections from -@code{buildbot/changes/hgbuildbot.py} run as an in-process 'changegroup' -hook) -@end itemize - -@item Arch/Bazaar -@itemize @bullet -@item pb.PBChangeSource (listening for connections from -@code{contrib/arch_buildbot.py} run in a commit hook) -@end itemize - -@item Bzr (the newer Bazaar) -@itemize @bullet -@item pb.PBChangeSource (listening for connections from -@code{contrib/bzr_buildbot.py} run in a post-change-branch-tip or commit hook) -@item @code{contrib/bzr_buildbot.py}'s BzrPoller (polling the Bzr repository) -@end itemize - -@item Git -@itemize @bullet -@item pb.PBChangeSource (listening for connections from -@code{contrib/git_buildbot.py} run in the post-receive hook) -@end itemize - -@end table - -All VC systems can be driven by a PBChangeSource and the -@code{buildbot sendchange} tool run from some form of commit script. -If you write an email parsing function, they can also all be driven by -a suitable @code{MaildirSource}. - - -@node Choosing ChangeSources, CVSToys - PBService, Change Sources, Getting Source Code Changes -@section Choosing ChangeSources - -The @code{master.cfg} configuration file has a dictionary key named -@code{BuildmasterConfig['change_source']}, which holds the active -@code{IChangeSource} object. The config file will typically create an -object from one of the classes described below and stuff it into this -key. - -Each buildmaster typically has just a single ChangeSource, since it is -only watching a single source tree. But if, for some reason, you need -multiple sources, just set @code{c['change_source']} to a list of -ChangeSources.. it will accept that too. - -@example -s = FreshCVSSourceNewcred(host="host", port=4519, - user="alice", passwd="secret", - prefix="Twisted") -BuildmasterConfig['change_source'] = [s] -@end example - -Each source tree has a nominal @code{top}. Each Change has a list of -filenames, which are all relative to this top location. The -ChangeSource is responsible for doing whatever is necessary to -accomplish this. Most sources have a @code{prefix} argument: a partial -pathname which is stripped from the front of all filenames provided to -that @code{ChangeSource}. Files which are outside this sub-tree are -ignored by the changesource: it does not generate Changes for those -files. - - -@node CVSToys - PBService, Mail-parsing ChangeSources, Choosing ChangeSources, Getting Source Code Changes -@section CVSToys - PBService - -@csindex buildbot.changes.freshcvs.FreshCVSSource - -The @uref{http://purl.net/net/CVSToys, CVSToys} package provides a -server which runs on the machine that hosts the CVS repository it -watches. It has a variety of ways to distribute commit notifications, -and offers a flexible regexp-based way to filter out uninteresting -changes. One of the notification options is named @code{PBService} and -works by listening on a TCP port for clients. These clients subscribe -to hear about commit notifications. - -The buildmaster has a CVSToys-compatible @code{PBService} client built -in. There are two versions of it, one for old versions of CVSToys -(1.0.9 and earlier) which used the @code{oldcred} authentication -framework, and one for newer versions (1.0.10 and later) which use -@code{newcred}. Both are classes in the -@code{buildbot.changes.freshcvs} package. - -@code{FreshCVSSourceNewcred} objects are created with the following -parameters: - -@table @samp - -@item @code{host} and @code{port} -these specify where the CVSToys server can be reached - -@item @code{user} and @code{passwd} -these specify the login information for the CVSToys server -(@code{freshcvs}). These must match the server's values, which are -defined in the @code{freshCfg} configuration file (which lives in the -CVSROOT directory of the repository). - -@item @code{prefix} -this is the prefix to be found and stripped from filenames delivered -by the CVSToys server. Most projects live in sub-directories of the -main repository, as siblings of the CVSROOT sub-directory, so -typically this prefix is set to that top sub-directory name. - -@end table - -@heading Example - -To set up the freshCVS server, add a statement like the following to -your @file{freshCfg} file: - -@example -pb = ConfigurationSet([ - (None, None, None, PBService(userpass=('foo', 'bar'), port=4519)), - ]) -@end example - -This will announce all changes to a client which connects to port 4519 -using a username of 'foo' and a password of 'bar'. - -Then add a clause like this to your buildmaster's @file{master.cfg}: - -@example -BuildmasterConfig['change_source'] = FreshCVSSource("cvs.example.com", 4519, - "foo", "bar", - prefix="glib/") -@end example - -where "cvs.example.com" is the host that is running the FreshCVS daemon, and -"glib" is the top-level directory (relative to the repository's root) where -all your source code lives. Most projects keep one or more projects in the -same repository (along with CVSROOT/ to hold admin files like loginfo and -freshCfg); the prefix= argument tells the buildmaster to ignore everything -outside that directory, and to strip that common prefix from all pathnames -it handles. - - -@node Mail-parsing ChangeSources, PBChangeSource, CVSToys - PBService, Getting Source Code Changes -@section Mail-parsing ChangeSources - -Many projects publish information about changes to their source tree -by sending an email message out to a mailing list, frequently named -PROJECT-commits or PROJECT-changes. Each message usually contains a -description of the change (who made the change, which files were -affected) and sometimes a copy of the diff. Humans can subscribe to -this list to stay informed about what's happening to the source tree. - -The Buildbot can also be subscribed to a -commits mailing list, and -can trigger builds in response to Changes that it hears about. The -buildmaster admin needs to arrange for these email messages to arrive -in a place where the buildmaster can find them, and configure the -buildmaster to parse the messages correctly. Once that is in place, -the email parser will create Change objects and deliver them to the -Schedulers (see @pxref{Change Sources and Schedulers}) just -like any other ChangeSource. - -There are two components to setting up an email-based ChangeSource. -The first is to route the email messages to the buildmaster, which is -done by dropping them into a ``maildir''. The second is to actually -parse the messages, which is highly dependent upon the tool that was -used to create them. Each VC system has a collection of favorite -change-emailing tools, and each has a slightly different format, so -each has a different parsing function. There is a separate -ChangeSource variant for each parsing function. - -Once you've chosen a maildir location and a parsing function, create -the change source and put it in @code{c['change_source']}: - -@example -from buildbot.changes.mail import SyncmailMaildirSource -c['change_source'] = SyncmailMaildirSource("~/maildir-buildbot", - prefix="/trunk/") -@end example - -@menu -* Subscribing the Buildmaster:: -* Using Maildirs:: -* Parsing Email Change Messages:: -@end menu - -@node Subscribing the Buildmaster, Using Maildirs, Mail-parsing ChangeSources, Mail-parsing ChangeSources -@subsection Subscribing the Buildmaster - -The recommended way to install the buildbot is to create a dedicated -account for the buildmaster. If you do this, the account will probably -have a distinct email address (perhaps -@email{buildmaster@@example.org}). Then just arrange for this -account's email to be delivered to a suitable maildir (described in -the next section). - -If the buildbot does not have its own account, ``extension addresses'' -can be used to distinguish between email intended for the buildmaster -and email intended for the rest of the account. In most modern MTAs, -the e.g. @code{foo@@example.org} account has control over every email -address at example.org which begins with "foo", such that email -addressed to @email{account-foo@@example.org} can be delivered to a -different destination than @email{account-bar@@example.org}. qmail -does this by using separate .qmail files for the two destinations -(@file{.qmail-foo} and @file{.qmail-bar}, with @file{.qmail} -controlling the base address and @file{.qmail-default} controlling all -other extensions). Other MTAs have similar mechanisms. - -Thus you can assign an extension address like -@email{foo-buildmaster@@example.org} to the buildmaster, and retain -@email{foo@@example.org} for your own use. - - -@node Using Maildirs, Parsing Email Change Messages, Subscribing the Buildmaster, Mail-parsing ChangeSources -@subsection Using Maildirs - -A ``maildir'' is a simple directory structure originally developed for -qmail that allows safe atomic update without locking. Create a base -directory with three subdirectories: ``new'', ``tmp'', and ``cur''. -When messages arrive, they are put into a uniquely-named file (using -pids, timestamps, and random numbers) in ``tmp''. When the file is -complete, it is atomically renamed into ``new''. Eventually the -buildmaster notices the file in ``new'', reads and parses the -contents, then moves it into ``cur''. A cronjob can be used to delete -files in ``cur'' at leisure. - -Maildirs are frequently created with the @command{maildirmake} tool, -but a simple @command{mkdir -p ~/MAILDIR/@{cur,new,tmp@}} is pretty much -equivalent. - -Many modern MTAs can deliver directly to maildirs. The usual .forward -or .procmailrc syntax is to name the base directory with a trailing -slash, so something like @code{~/MAILDIR/} . qmail and postfix are -maildir-capable MTAs, and procmail is a maildir-capable MDA (Mail -Delivery Agent). - -For MTAs which cannot put files into maildirs directly, the -``safecat'' tool can be executed from a .forward file to accomplish -the same thing. - -The Buildmaster uses the linux DNotify facility to receive immediate -notification when the maildir's ``new'' directory has changed. When -this facility is not available, it polls the directory for new -messages, every 10 seconds by default. - -@node Parsing Email Change Messages, , Using Maildirs, Mail-parsing ChangeSources -@subsection Parsing Email Change Messages - -The second component to setting up an email-based ChangeSource is to -parse the actual notices. This is highly dependent upon the VC system -and commit script in use. - -A couple of common tools used to create these change emails are: - -@table @samp - -@item CVS -@table @samp -@item CVSToys MailNotifier -@ref{FCMaildirSource} -@item Bonsai notification -@ref{BonsaiMaildirSource} -@item syncmail -@ref{SyncmailMaildirSource} -@end table - -@item SVN -@table @samp -@item svnmailer -http://opensource.perlig.de/en/svnmailer/ -@item commit-email.pl -@ref{SVNCommitEmailMaildirSource} -@end table - -@item Bzr -@table @samp -@item Launchpad -@ref{BzrLaunchpadEmailMaildirSource} -@end table - -@item Mercurial -@table @samp -@item NotifyExtension -http://www.selenic.com/mercurial/wiki/index.cgi/NotifyExtension -@end table - -@item Git -@table @samp -@item post-receive-email -http://git.kernel.org/?p=git/git.git;a=blob;f=contrib/hooks/post-receive-email;hb=HEAD -@end table - -@end table - - -The following sections describe the parsers available for each of -these tools. - -Most of these parsers accept a @code{prefix=} argument, which is used -to limit the set of files that the buildmaster pays attention to. This -is most useful for systems like CVS and SVN which put multiple -projects in a single repository (or use repository names to indicate -branches). Each filename that appears in the email is tested against -the prefix: if the filename does not start with the prefix, the file -is ignored. If the filename @emph{does} start with the prefix, that -prefix is stripped from the filename before any further processing is -done. Thus the prefix usually ends with a slash. - -@menu -* FCMaildirSource:: -* SyncmailMaildirSource:: -* BonsaiMaildirSource:: -* SVNCommitEmailMaildirSource:: -* BzrLaunchpadEmailMaildirSource:: -@end menu - -@node FCMaildirSource, SyncmailMaildirSource, Parsing Email Change Messages, Parsing Email Change Messages -@subsubsection FCMaildirSource - - -@csindex buildbot.changes.mail.FCMaildirSource - -http://twistedmatrix.com/users/acapnotic/wares/code/CVSToys/ - -This parser works with the CVSToys @code{MailNotification} action, -which will send email to a list of recipients for each commit. This -tends to work better than using @code{/bin/mail} from within the -CVSROOT/loginfo file directly, as CVSToys will batch together all -files changed during the same CVS invocation, and can provide more -information (like creating a ViewCVS URL for each file changed). - -The Buildbot's @code{FCMaildirSource} knows for to parse these CVSToys -messages and turn them into Change objects. It can be given two -parameters: the directory name of the maildir root, and the prefix to -strip. - -@example -from buildbot.changes.mail import FCMaildirSource -c['change_source'] = FCMaildirSource("~/maildir-buildbot") -@end example - -@node SyncmailMaildirSource, BonsaiMaildirSource, FCMaildirSource, Parsing Email Change Messages -@subsubsection SyncmailMaildirSource - -@csindex buildbot.changes.mail.SyncmailMaildirSource - -http://sourceforge.net/projects/cvs-syncmail - -@code{SyncmailMaildirSource} knows how to parse the message format used by -the CVS ``syncmail'' script. - -@example -from buildbot.changes.mail import SyncmailMaildirSource -c['change_source'] = SyncmailMaildirSource("~/maildir-buildbot") -@end example - -@node BonsaiMaildirSource, SVNCommitEmailMaildirSource, SyncmailMaildirSource, Parsing Email Change Messages -@subsubsection BonsaiMaildirSource - -@csindex buildbot.changes.mail.BonsaiMaildirSource - -http://www.mozilla.org/bonsai.html - -@code{BonsaiMaildirSource} parses messages sent out by Bonsai, the CVS -tree-management system built by Mozilla. - -@example -from buildbot.changes.mail import BonsaiMaildirSource -c['change_source'] = BonsaiMaildirSource("~/maildir-buildbot") -@end example - -@node SVNCommitEmailMaildirSource, BzrLaunchpadEmailMaildirSource, BonsaiMaildirSource, Parsing Email Change Messages -@subsubsection SVNCommitEmailMaildirSource - -@csindex buildbot.changes.mail.SVNCommitEmailMaildirSource - -@code{SVNCommitEmailMaildirSource} parses message sent out by the -@code{commit-email.pl} script, which is included in the Subversion -distribution. - -It does not currently handle branches: all of the Change objects that -it creates will be associated with the default (i.e. trunk) branch. - -@example -from buildbot.changes.mail import SVNCommitEmailMaildirSource -c['change_source'] = SVNCommitEmailMaildirSource("~/maildir-buildbot") -@end example - -@node BzrLaunchpadEmailMaildirSource, , SVNCommitEmailMaildirSource, Parsing Email Change Messages -@subsubsection BzrLaunchpadEmailMaildirSource - -@csindex buildbot.changes.mail.BzrLaunchpadEmailMaildirSource - -@code{BzrLaunchpadEmailMaildirSource} parses the mails that are sent to -addresses that subscribe to branch revision notifications for a bzr branch -hosted on Launchpad. - -The branch name defaults to @code{lp:}. For example -@code{lp:~maria-captains/maria/5.1}. - -If only a single branch is used, the default branch name can be changed by -setting @code{defaultBranch}. - -For multiple branches, pass a dictionary as the value of the @code{branchMap} -option to map specific repository paths to specific branch names (see example -below). The leading @code{lp:} prefix of the path is optional. - -The @code{prefix} option is not supported (it is silently ignored). Use the -@code{branchMap} and @code{defaultBranch} instead to assign changes to -branches (and just do not subscribe the buildbot to branches that are not of -interest). - -The revision number is obtained from the email text. The bzr revision id is -not available in the mails sent by Launchpad. However, it is possible to set -the bzr @code{append_revisions_only} option for public shared repositories to -avoid new pushes of merges changing the meaning of old revision numbers. - -@verbatim -from buildbot.changes.mail import BzrLaunchpadEmailMaildirSource -bm = { 'lp:~maria-captains/maria/5.1' : '5.1', 'lp:~maria-captains/maria/6.0' : '6.0' } -c['change_source'] = BzrLaunchpadEmailMaildirSource("~/maildir-buildbot", branchMap = bm) -@end verbatim - - -@node PBChangeSource, P4Source, Mail-parsing ChangeSources, Getting Source Code Changes -@section PBChangeSource - -@csindex buildbot.changes.pb.PBChangeSource - -The last kind of ChangeSource actually listens on a TCP port for -clients to connect and push change notices @emph{into} the -Buildmaster. This is used by the built-in @code{buildbot sendchange} -notification tool, as well as the VC-specific -@file{contrib/svn_buildbot.py}, @file{contrib/arch_buildbot.py}, -@file{contrib/hg_buildbot.py} tools, and the -@code{buildbot.changes.hgbuildbot} hook. These tools are run by the -repository (in a commit hook script), and connect to the buildmaster -directly each time a file is comitted. This is also useful for -creating new kinds of change sources that work on a @code{push} model -instead of some kind of subscription scheme, for example a script -which is run out of an email .forward file. - -This ChangeSource can be configured to listen on its own TCP port, or -it can share the port that the buildmaster is already using for the -buildslaves to connect. (This is possible because the -@code{PBChangeSource} uses the same protocol as the buildslaves, and -they can be distinguished by the @code{username} attribute used when -the initial connection is established). It might be useful to have it -listen on a different port if, for example, you wanted to establish -different firewall rules for that port. You could allow only the SVN -repository machine access to the @code{PBChangeSource} port, while -allowing only the buildslave machines access to the slave port. Or you -could just expose one port and run everything over it. @emph{Note: -this feature is not yet implemented, the PBChangeSource will always -share the slave port and will always have a @code{user} name of -@code{change}, and a passwd of @code{changepw}. These limitations will -be removed in the future.}. - - -The @code{PBChangeSource} is created with the following arguments. All -are optional. - -@table @samp -@item @code{port} -which port to listen on. If @code{None} (which is the default), it -shares the port used for buildslave connections. @emph{Not -Implemented, always set to @code{None}}. - -@item @code{user} and @code{passwd} -The user/passwd account information that the client program must use -to connect. Defaults to @code{change} and @code{changepw}. @emph{Not -Implemented, @code{user} is currently always set to @code{change}, -@code{passwd} is always set to @code{changepw}}. - -@item @code{prefix} -The prefix to be found and stripped from filenames delivered over the -connection. Any filenames which do not start with this prefix will be -removed. If all the filenames in a given Change are removed, the that -whole Change will be dropped. This string should probably end with a -directory separator. - -This is useful for changes coming from version control systems that -represent branches as parent directories within the repository (like -SVN and Perforce). Use a prefix of 'trunk/' or -'project/branches/foobranch/' to only follow one branch and to get -correct tree-relative filenames. Without a prefix, the PBChangeSource -will probably deliver Changes with filenames like @file{trunk/foo.c} -instead of just @file{foo.c}. Of course this also depends upon the -tool sending the Changes in (like @command{buildbot sendchange}) and -what filenames it is delivering: that tool may be filtering and -stripping prefixes at the sending end. - -@end table - -@node P4Source, BonsaiPoller, PBChangeSource, Getting Source Code Changes -@section P4Source - -@csindex buildbot.changes.p4poller.P4Source - -The @code{P4Source} periodically polls a @uref{http://www.perforce.com/, -Perforce} depot for changes. It accepts the following arguments: - -@table @samp -@item @code{p4base} -The base depot path to watch, without the trailing '/...'. - -@item @code{p4port} -The Perforce server to connect to (as host:port). - -@item @code{p4user} -The Perforce user. - -@item @code{p4passwd} -The Perforce password. - -@item @code{p4bin} -An optional string parameter. Specify the location of the perforce command -line binary (p4). You only need to do this if the perforce binary is not -in the path of the buildbot user. Defaults to ``p4''. - -@item @code{split_file} -A function that maps a pathname, without the leading @code{p4base}, to a -(branch, filename) tuple. The default just returns (None, branchfile), -which effectively disables branch support. You should supply a function -which understands your repository structure. - -@item @code{pollinterval} -How often to poll, in seconds. Defaults to 600 (10 minutes). - -@item @code{histmax} -The maximum number of changes to inspect at a time. If more than this -number occur since the last poll, older changes will be silently -ignored. -@end table - -@heading Example - -This configuration uses the @code{P4PORT}, @code{P4USER}, and @code{P4PASSWD} -specified in the buildmaster's environment. It watches a project in which the -branch name is simply the next path component, and the file is all path -components after. - -@example -from buildbot.changes import p4poller -s = p4poller.P4Source(p4base='//depot/project/', - split_file=lambda branchfile: branchfile.split('/',1), - ) -c['change_source'] = s -@end example - -@node BonsaiPoller, SVNPoller, P4Source, Getting Source Code Changes -@section BonsaiPoller - -@csindex buildbot.changes.bonsaipoller.BonsaiPoller - -The @code{BonsaiPoller} periodically polls a Bonsai server. This is a -CGI script accessed through a web server that provides information -about a CVS tree, for example the Mozilla bonsai server at -@uref{http://bonsai.mozilla.org}. Bonsai servers are usable by both -humans and machines. In this case, the buildbot's change source forms -a query which asks about any files in the specified branch which have -changed since the last query. - -Please take a look at the BonsaiPoller docstring for details about the -arguments it accepts. - - -@node SVNPoller, MercurialHook, BonsaiPoller, Getting Source Code Changes -@section SVNPoller - -@csindex buildbot.changes.svnpoller.SVNPoller - -The @code{buildbot.changes.svnpoller.SVNPoller} is a ChangeSource -which periodically polls a @uref{http://subversion.tigris.org/, -Subversion} repository for new revisions, by running the @code{svn -log} command in a subshell. It can watch a single branch or multiple -branches. - -@code{SVNPoller} accepts the following arguments: - -@table @code -@item svnurl -The base URL path to watch, like -@code{svn://svn.twistedmatrix.com/svn/Twisted/trunk}, or -@code{http://divmod.org/svn/Divmod/}, or even -@code{file:///home/svn/Repository/ProjectA/branches/1.5/}. This must -include the access scheme, the location of the repository (both the -hostname for remote ones, and any additional directory names necessary -to get to the repository), and the sub-path within the repository's -virtual filesystem for the project and branch of interest. - -The @code{SVNPoller} will only pay attention to files inside the -subdirectory specified by the complete svnurl. - -@item split_file -A function to convert pathnames into (branch, relative_pathname) -tuples. Use this to explain your repository's branch-naming policy to -@code{SVNPoller}. This function must accept a single string and return -a two-entry tuple. There are a few utility functions in -@code{buildbot.changes.svnpoller} that can be used as a -@code{split_file} function, see below for details. - -The default value always returns (None, path), which indicates that -all files are on the trunk. - -Subclasses of @code{SVNPoller} can override the @code{split_file} -method instead of using the @code{split_file=} argument. - -@item svnuser -An optional string parameter. If set, the @code{--user} argument will -be added to all @code{svn} commands. Use this if you have to -authenticate to the svn server before you can do @code{svn info} or -@code{svn log} commands. - -@item svnpasswd -Like @code{svnuser}, this will cause a @code{--password} argument to -be passed to all svn commands. - -@item pollinterval -How often to poll, in seconds. Defaults to 600 (checking once every 10 -minutes). Lower this if you want the buildbot to notice changes -faster, raise it if you want to reduce the network and CPU load on -your svn server. Please be considerate of public SVN repositories by -using a large interval when polling them. - -@item histmax -The maximum number of changes to inspect at a time. Every POLLINTERVAL -seconds, the @code{SVNPoller} asks for the last HISTMAX changes and -looks through them for any ones it does not already know about. If -more than HISTMAX revisions have been committed since the last poll, -older changes will be silently ignored. Larger values of histmax will -cause more time and memory to be consumed on each poll attempt. -@code{histmax} defaults to 100. - -@item svnbin -This controls the @code{svn} executable to use. If subversion is -installed in a weird place on your system (outside of the -buildmaster's @code{$PATH}), use this to tell @code{SVNPoller} where -to find it. The default value of ``svn'' will almost always be -sufficient. - -@item revlinktmpl -This parameter allows a link to be provided for each revision (for example, -to websvn or viewvc). These links appear anywhere changes are shown, such -as on build or change pages. The proper form for this parameter is an URL -with the portion that will substitute for a revision number replaced by -''%s''. For example, @code{'http://myserver/websvn/revision.php?rev=%s'} -could be used to cause revision links to be created to a websvn repository -viewer. - -@end table - -@heading Branches - -Each source file that is tracked by a Subversion repository has a -fully-qualified SVN URL in the following form: -(REPOURL)(PROJECT-plus-BRANCH)(FILEPATH). When you create the -@code{SVNPoller}, you give it a @code{svnurl} value that includes all -of the REPOURL and possibly some portion of the PROJECT-plus-BRANCH -string. The @code{SVNPoller} is responsible for producing Changes that -contain a branch name and a FILEPATH (which is relative to the top of -a checked-out tree). The details of how these strings are split up -depend upon how your repository names its branches. - -@subheading PROJECT/BRANCHNAME/FILEPATH repositories - -One common layout is to have all the various projects that share a -repository get a single top-level directory each. Then under a given -project's directory, you get two subdirectories, one named ``trunk'' -and another named ``branches''. Under ``branches'' you have a bunch of -other directories, one per branch, with names like ``1.5.x'' and -``testing''. It is also common to see directories like ``tags'' and -``releases'' next to ``branches'' and ``trunk''. - -For example, the Twisted project has a subversion server on -``svn.twistedmatrix.com'' that hosts several sub-projects. The -repository is available through a SCHEME of ``svn:''. The primary -sub-project is Twisted, of course, with a repository root of -``svn://svn.twistedmatrix.com/svn/Twisted''. Another sub-project is -Informant, with a root of -``svn://svn.twistedmatrix.com/svn/Informant'', etc. Inside any -checked-out Twisted tree, there is a file named bin/trial (which is -used to run unit test suites). - -The trunk for Twisted is in -``svn://svn.twistedmatrix.com/svn/Twisted/trunk'', and the -fully-qualified SVN URL for the trunk version of @code{trial} would be -``svn://svn.twistedmatrix.com/svn/Twisted/trunk/bin/trial''. The same -SVNURL for that file on a branch named ``1.5.x'' would be -``svn://svn.twistedmatrix.com/svn/Twisted/branches/1.5.x/bin/trial''. - -To set up a @code{SVNPoller} that watches the Twisted trunk (and -nothing else), we would use the following: - -@example -from buildbot.changes.svnpoller import SVNPoller -c['change_source'] = SVNPoller("svn://svn.twistedmatrix.com/svn/Twisted/trunk") -@end example - -In this case, every Change that our @code{SVNPoller} produces will -have @code{.branch=None}, to indicate that the Change is on the trunk. -No other sub-projects or branches will be tracked. - -If we want our ChangeSource to follow multiple branches, we have to do -two things. First we have to change our @code{svnurl=} argument to -watch more than just ``.../Twisted/trunk''. We will set it to -``.../Twisted'' so that we'll see both the trunk and all the branches. -Second, we have to tell @code{SVNPoller} how to split the -(PROJECT-plus-BRANCH)(FILEPATH) strings it gets from the repository -out into (BRANCH) and (FILEPATH) pairs. - -We do the latter by providing a ``split_file'' function. This function -is responsible for splitting something like -``branches/1.5.x/bin/trial'' into @code{branch}=''branches/1.5.x'' and -@code{filepath}=''bin/trial''. This function is always given a string -that names a file relative to the subdirectory pointed to by the -@code{SVNPoller}'s @code{svnurl=} argument. It is expected to return a -(BRANCHNAME, FILEPATH) tuple (in which FILEPATH is relative to the -branch indicated), or None to indicate that the file is outside any -project of interest. - -(note that we want to see ``branches/1.5.x'' rather than just -``1.5.x'' because when we perform the SVN checkout, we will probably -append the branch name to the baseURL, which requires that we keep the -``branches'' component in there. Other VC schemes use a different -approach towards branches and may not require this artifact.) - -If your repository uses this same PROJECT/BRANCH/FILEPATH naming -scheme, the following function will work: - -@example -def split_file_branches(path): - pieces = path.split('/') - if pieces[0] == 'trunk': - return (None, '/'.join(pieces[1:])) - elif pieces[0] == 'branches': - return ('/'.join(pieces[0:2]), - '/'.join(pieces[2:])) - else: - return None -@end example - -This function is provided as -@code{buildbot.changes.svnpoller.split_file_branches} for your -convenience. So to have our Twisted-watching @code{SVNPoller} follow -multiple branches, we would use this: - -@example -from buildbot.changes.svnpoller import SVNPoller, split_file_branches -c['change_source'] = SVNPoller("svn://svn.twistedmatrix.com/svn/Twisted", - split_file=split_file_branches) -@end example - -Changes for all sorts of branches (with names like ``branches/1.5.x'', -and None to indicate the trunk) will be delivered to the Schedulers. -Each Scheduler is then free to use or ignore each branch as it sees -fit. - -@subheading BRANCHNAME/PROJECT/FILEPATH repositories - -Another common way to organize a Subversion repository is to put the -branch name at the top, and the projects underneath. This is -especially frequent when there are a number of related sub-projects -that all get released in a group. - -For example, Divmod.org hosts a project named ``Nevow'' as well as one -named ``Quotient''. In a checked-out Nevow tree there is a directory -named ``formless'' that contains a python source file named -``webform.py''. This repository is accessible via webdav (and thus -uses an ``http:'' scheme) through the divmod.org hostname. There are -many branches in this repository, and they use a -(BRANCHNAME)/(PROJECT) naming policy. - -The fully-qualified SVN URL for the trunk version of webform.py is -@code{http://divmod.org/svn/Divmod/trunk/Nevow/formless/webform.py}. -You can do an @code{svn co} with that URL and get a copy of the latest -version. The 1.5.x branch version of this file would have a URL of -@code{http://divmod.org/svn/Divmod/branches/1.5.x/Nevow/formless/webform.py}. -The whole Nevow trunk would be checked out with -@code{http://divmod.org/svn/Divmod/trunk/Nevow}, while the Quotient -trunk would be checked out using -@code{http://divmod.org/svn/Divmod/trunk/Quotient}. - -Now suppose we want to have an @code{SVNPoller} that only cares about -the Nevow trunk. This case looks just like the PROJECT/BRANCH layout -described earlier: - -@example -from buildbot.changes.svnpoller import SVNPoller -c['change_source'] = SVNPoller("http://divmod.org/svn/Divmod/trunk/Nevow") -@end example - -But what happens when we want to track multiple Nevow branches? We -have to point our @code{svnurl=} high enough to see all those -branches, but we also don't want to include Quotient changes (since -we're only building Nevow). To accomplish this, we must rely upon the -@code{split_file} function to help us tell the difference between -files that belong to Nevow and those that belong to Quotient, as well -as figuring out which branch each one is on. - -@example -from buildbot.changes.svnpoller import SVNPoller -c['change_source'] = SVNPoller("http://divmod.org/svn/Divmod", - split_file=my_file_splitter) -@end example - -The @code{my_file_splitter} function will be called with -repository-relative pathnames like: - -@table @code -@item trunk/Nevow/formless/webform.py -This is a Nevow file, on the trunk. We want the Change that includes this -to see a filename of @code{formless/webform.py"}, and a branch of None - -@item branches/1.5.x/Nevow/formless/webform.py -This is a Nevow file, on a branch. We want to get -branch=''branches/1.5.x'' and filename=''formless/webform.py''. - -@item trunk/Quotient/setup.py -This is a Quotient file, so we want to ignore it by having -@code{my_file_splitter} return None. - -@item branches/1.5.x/Quotient/setup.py -This is also a Quotient file, which should be ignored. -@end table - -The following definition for @code{my_file_splitter} will do the job: - -@example -def my_file_splitter(path): - pieces = path.split('/') - if pieces[0] == 'trunk': - branch = None - pieces.pop(0) # remove 'trunk' - elif pieces[0] == 'branches': - pieces.pop(0) # remove 'branches' - # grab branch name - branch = 'branches/' + pieces.pop(0) - else: - return None # something weird - projectname = pieces.pop(0) - if projectname != 'Nevow': - return None # wrong project - return (branch, '/'.join(pieces)) -@end example - -@node MercurialHook, Bzr Hook, SVNPoller, Getting Source Code Changes -@section MercurialHook - -Since Mercurial is written in python, the hook script can invoke -Buildbot's @code{sendchange} function directly, rather than having to -spawn an external process. This function delivers the same sort of -changes as @code{buildbot sendchange} and the various hook scripts in -contrib/, so you'll need to add a @code{pb.PBChangeSource} to your -buildmaster to receive these changes. - -To set this up, first choose a Mercurial repository that represents -your central ``official'' source tree. This will be the same -repository that your buildslaves will eventually pull from. Install -Buildbot on the machine that hosts this repository, using the same -version of python as Mercurial is using (so that the Mercurial hook -can import code from buildbot). Then add the following to the -@code{.hg/hgrc} file in that repository, replacing the buildmaster -hostname/portnumber as appropriate for your buildbot: - -@example -[hooks] -changegroup.buildbot = python:buildbot.changes.hgbuildbot.hook - -[hgbuildbot] -master = buildmaster.example.org:9987 -@end example - -(Note that Mercurial lets you define multiple @code{changegroup} hooks -by giving them distinct names, like @code{changegroup.foo} and -@code{changegroup.bar}, which is why we use -@code{changegroup.buildbot} in this example. There is nothing magical -about the ``buildbot'' suffix in the hook name. The -@code{[hgbuildbot]} section @emph{is} special, however, as it is the -only section that the buildbot hook pays attention to.) - -Also note that this runs as a @code{changegroup} hook, rather than as -an @code{incoming} hook. The @code{changegroup} hook is run with -multiple revisions at a time (say, if multiple revisions are being -pushed to this repository in a single @command{hg push} command), -whereas the @code{incoming} hook is run with just one revision at a -time. The @code{hgbuildbot.hook} function will only work with the -@code{changegroup} hook. - -The @code{[hgbuildbot]} section has two other parameters that you -might specify, both of which control the name of the branch that is -attached to the changes coming from this hook. - -One common branch naming policy for Mercurial repositories is to use -it just like Darcs: each branch goes into a separate repository, and -all the branches for a single project share a common parent directory. -For example, you might have @file{/var/repos/PROJECT/trunk/} and -@file{/var/repos/PROJECT/release}. To use this style, use the -@code{branchtype = dirname} setting, which simply uses the last -component of the repository's enclosing directory as the branch name: - -@example -[hgbuildbot] -master = buildmaster.example.org:9987 -branchtype = dirname -@end example - -Another approach is to use Mercurial's built-in branches (the kind -created with @command{hg branch} and listed with @command{hg -branches}). This feature associates persistent names with particular -lines of descent within a single repository. (note that the buildbot -@code{source.Mercurial} checkout step does not yet support this kind -of branch). To have the commit hook deliver this sort of branch name -with the Change object, use @code{branchtype = inrepo}: - -@example -[hgbuildbot] -master = buildmaster.example.org:9987 -branchtype = inrepo -@end example - -Finally, if you want to simply specify the branchname directly, for -all changes, use @code{branch = BRANCHNAME}. This overrides -@code{branchtype}: - -@example -[hgbuildbot] -master = buildmaster.example.org:9987 -branch = trunk -@end example - -If you use @code{branch=} like this, you'll need to put a separate -.hgrc in each repository. If you use @code{branchtype=}, you may be -able to use the same .hgrc for all your repositories, stored in -@file{~/.hgrc} or @file{/etc/mercurial/hgrc}. - - -@node Bzr Hook, Bzr Poller, MercurialHook, Getting Source Code Changes -@section Bzr Hook - -Bzr is also written in Python, and the Bzr hook depends on Twisted to send the -changes. - -To install, put @code{contrib/bzr_buildbot.py} in one of your plugins -locations a bzr plugins directory (e.g., -@code{~/.bazaar/plugins}). Then, in one of your bazaar conf files (e.g., -@code{~/.bazaar/locations.conf}), set the location you want to connect with buildbot -with these keys: - -@table @code -@item buildbot_on -one of 'commit', 'push, or 'change'. Turns the plugin on to report changes via -commit, changes via push, or any changes to the trunk. 'change' is -recommended. - -@item buildbot_server -(required to send to a buildbot master) the URL of the buildbot master to -which you will connect (as of this writing, the same server and port to which -slaves connect). - -@item buildbot_port -(optional, defaults to 9989) the port of the buildbot master to which you will -connect (as of this writing, the same server and port to which slaves connect) - -@item buildbot_pqm -(optional, defaults to not pqm) Normally, the user that commits the revision -is the user that is responsible for the change. When run in a pqm (Patch Queue -Manager, see https://launchpad.net/pqm) environment, the user that commits is -the Patch Queue Manager, and the user that committed the *parent* revision is -responsible for the change. To turn on the pqm mode, set this value to any of -(case-insensitive) "Yes", "Y", "True", or "T". - -@item buildbot_dry_run -(optional, defaults to not a dry run) Normally, the post-commit hook will -attempt to communicate with the configured buildbot server and port. If this -parameter is included and any of (case-insensitive) "Yes", "Y", "True", or -"T", then the hook will simply print what it would have sent, but not attempt -to contact the buildbot master. - -@item buildbot_send_branch_name -(optional, defaults to not sending the branch name) If your buildbot's bzr -source build step uses a repourl, do *not* turn this on. If your buildbot's -bzr build step uses a baseURL, then you may set this value to any of -(case-insensitive) "Yes", "Y", "True", or "T" to have the buildbot master -append the branch name to the baseURL. - -@end table - -When buildbot no longer has a hardcoded password, it will be a configuration -option here as well. - -Here's a simple example that you might have in your -@code{~/.bazaar/locations.conf}. - -@example -[chroot-*:///var/local/myrepo/mybranch] -buildbot_on = change -buildbot_server = localhost -@end example - -@node Bzr Poller, , Bzr Hook, Getting Source Code Changes -@section Bzr Poller - -If you cannot insert a Bzr hook in the server, you can use the Bzr Poller. To -use, put @code{contrib/bzr_buildbot.py} somewhere that your buildbot -configuration can import it. Even putting it in the same directory as the master.cfg -should work. Install the poller in the buildbot configuration as with any -other change source. Minimally, provide a URL that you want to poll (bzr://, -bzr+ssh://, or lp:), though make sure the buildbot user has necessary -privileges. You may also want to specify these optional values. - -@table @code -@item poll_interval -The number of seconds to wait between polls. Defaults to 10 minutes. - -@item branch_name -Any value to be used as the branch name. Defaults to None, or specify a -string, or specify the constants from @code{bzr_buildbot.py} SHORT or FULL to -get the short branch name or full branch address. - -@item blame_merge_author -normally, the user that commits the revision is the user that is responsible -for the change. When run in a pqm (Patch Queue Manager, see -https://launchpad.net/pqm) environment, the user that commits is the Patch -Queue Manager, and the user that committed the merged, *parent* revision is -responsible for the change. set this value to True if this is pointed against -a PQM-managed branch. -@end table - -@node Build Process, Status Delivery, Getting Source Code Changes, Top -@chapter Build Process - -A @code{Build} object is responsible for actually performing a build. -It gets access to a remote @code{SlaveBuilder} where it may run -commands, and a @code{BuildStatus} object where it must emit status -events. The @code{Build} is created by the Builder's -@code{BuildFactory}. - -The default @code{Build} class is made up of a fixed sequence of -@code{BuildSteps}, executed one after another until all are complete -(or one of them indicates that the build should be halted early). The -default @code{BuildFactory} creates instances of this @code{Build} -class with a list of @code{BuildSteps}, so the basic way to configure -the build is to provide a list of @code{BuildSteps} to your -@code{BuildFactory}. - -More complicated @code{Build} subclasses can make other decisions: -execute some steps only if certain files were changed, or if certain -previous steps passed or failed. The base class has been written to -allow users to express basic control flow without writing code, but -you can always subclass and customize to achieve more specialized -behavior. - -@menu -* Build Steps:: -* Interlocks:: -* Build Factories:: -@end menu - -@node Build Steps, Interlocks, Build Process, Build Process -@section Build Steps - -@code{BuildStep}s are usually specified in the buildmaster's -configuration file, in a list that goes into the @code{BuildFactory}. -The @code{BuildStep} instances in this list are used as templates to -construct new independent copies for each build (so that state can be -kept on the @code{BuildStep} in one build without affecting a later -build). Each @code{BuildFactory} can be created with a list of steps, -or the factory can be created empty and then steps added to it using -the @code{addStep} method: - -@example -from buildbot.steps import source, shell -from buildbot.process import factory - -f = factory.BuildFactory() -f.addStep(source.SVN(svnurl="http://svn.example.org/Trunk/")) -f.addStep(shell.ShellCommand(command=["make", "all"])) -f.addStep(shell.ShellCommand(command=["make", "test"])) -@end example - -In earlier versions (0.7.5 and older), these steps were specified with -a tuple of (step_class, keyword_arguments). Steps can still be -specified this way, but the preferred form is to pass actual -@code{BuildStep} instances to @code{addStep}, because that gives the -@code{BuildStep} class a chance to do some validation on the -arguments. - -If you have a common set of steps which are used in several factories, the -@code{addSteps} method may be handy. It takes an iterable of @code{BuildStep} -instances. - -@example -setup_steps = [ - source.SVN(svnurl="http://svn.example.org/Trunk/") - shell.ShellCommand(command="./setup") -] -quick = factory.BuildFactory() -quick.addSteps(setup_steps) -quick.addStep(shell.shellCommand(command="make quick")) -@end example - -The rest of this section lists all the standard BuildStep objects -available for use in a Build, and the parameters which can be used to -control each. - -@menu -* Common Parameters:: -* Using Build Properties:: -* Source Checkout:: -* ShellCommand:: -* Simple ShellCommand Subclasses:: -* Python BuildSteps:: -* Transferring Files:: -* Steps That Run on the Master:: -* Triggering Schedulers:: -* Writing New BuildSteps:: -@end menu - -@node Common Parameters, Using Build Properties, Build Steps, Build Steps -@subsection Common Parameters - -The standard @code{Build} runs a series of @code{BuildStep}s in order, -only stopping when it runs out of steps or if one of them requests -that the build be halted. It collects status information from each one -to create an overall build status (of SUCCESS, WARNINGS, or FAILURE). - -All BuildSteps accept some common parameters. Some of these control -how their individual status affects the overall build. Others are used -to specify which @code{Locks} (see @pxref{Interlocks}) should be -acquired before allowing the step to run. - -Arguments common to all @code{BuildStep} subclasses: - - -@table @code -@item name -the name used to describe the step on the status display. It is also -used to give a name to any LogFiles created by this step. - -@item haltOnFailure -if True, a FAILURE of this build step will cause the build to halt -immediately. Steps with @code{alwaysRun=True} are still run. Generally -speaking, haltOnFailure implies flunkOnFailure (the default for most -BuildSteps). In some cases, particularly series of tests, it makes sense -to haltOnFailure if something fails early on but not flunkOnFailure. -This can be achieved with haltOnFailure=True, flunkOnFailure=False. - -@item flunkOnWarnings -when True, a WARNINGS or FAILURE of this build step will mark the -overall build as FAILURE. The remaining steps will still be executed. - -@item flunkOnFailure -when True, a FAILURE of this build step will mark the overall build as -a FAILURE. The remaining steps will still be executed. - -@item warnOnWarnings -when True, a WARNINGS or FAILURE of this build step will mark the -overall build as having WARNINGS. The remaining steps will still be -executed. - -@item warnOnFailure -when True, a FAILURE of this build step will mark the overall build as -having WARNINGS. The remaining steps will still be executed. - -@item alwaysRun -if True, this build step will always be run, even if a previous buildstep -with @code{haltOnFailure=True} has failed. - -@item doStepIf -A step can be configured to only run under certain conditions. To do this, set -the step's @code{doStepIf} to a boolean value, or to a function that returns a -boolean value. If the value or function result is false, then the step will -return SKIPPED without doing anything. Oherwise, the step will be executed -normally. If you set @code{doStepIf} to a function, that function should -accept one parameter, which will be the @code{Step} object itself. - -@item locks -a list of Locks (instances of @code{buildbot.locks.SlaveLock} or -@code{buildbot.locks.MasterLock}) that should be acquired before -starting this Step. The Locks will be released when the step is -complete. Note that this is a list of actual Lock instances, not -names. Also note that all Locks must have unique names. - -@end table - -@node Using Build Properties, Source Checkout, Common Parameters, Build Steps -@subsection Using Build Properties -@cindex Properties - -Build properties are a generalized way to provide configuration -information to build steps; see @ref{Build Properties}. - -Some build properties are inherited from external sources -- global -properties, schedulers, or buildslaves. Some build properties are -set when the build starts, such as the SourceStamp information. Other -properties can be set by BuildSteps as they run, for example the -various Source steps will set the @code{got_revision} property to the -source revision that was actually checked out (which can be useful -when the SourceStamp in use merely requested the ``latest revision'': -@code{got_revision} will tell you what was actually built). - -In custom BuildSteps, you can get and set the build properties with -the @code{getProperty}/@code{setProperty} methods. Each takes a string -for the name of the property, and returns or accepts an -arbitrary@footnote{Build properties are serialized along with the -build results, so they must be serializable. For this reason, the -value of any build property should be simple inert data: strings, -numbers, lists, tuples, and dictionaries. They should not contain -class instances.} object. For example: - -@example -class MakeTarball(ShellCommand): - def start(self): - if self.getProperty("os") == "win": - self.setCommand([ ... ]) # windows-only command - else: - self.setCommand([ ... ]) # equivalent for other systems - ShellCommand.start(self) -@end example - -@heading WithProperties -@cindex WithProperties - -You can use build properties in ShellCommands by using the -@code{WithProperties} wrapper when setting the arguments of -the ShellCommand. This interpolates the named build properties -into the generated shell command. Most step parameters accept -@code{WithProperties}. Please file bugs for any parameters which -do not. - -@example -from buildbot.steps.shell import ShellCommand -from buildbot.process.properties import WithProperties - -f.addStep(ShellCommand( - command=["tar", "czf", - WithProperties("build-%s.tar.gz", "revision"), - "source"])) -@end example - -If this BuildStep were used in a tree obtained from Subversion, it -would create a tarball with a name like @file{build-1234.tar.gz}. - -The @code{WithProperties} function does @code{printf}-style string -interpolation, using strings obtained by calling -@code{build.getProperty(propname)}. Note that for every @code{%s} (or -@code{%d}, etc), you must have exactly one additional argument to -indicate which build property you want to insert. - -You can also use python dictionary-style string interpolation by using -the @code{%(propname)s} syntax. In this form, the property name goes -in the parentheses, and WithProperties takes @emph{no} additional -arguments: - -@example -f.addStep(ShellCommand( - command=["tar", "czf", - WithProperties("build-%(revision)s.tar.gz"), - "source"])) -@end example - -Don't forget the extra ``s'' after the closing parenthesis! This is -the cause of many confusing errors. - -The dictionary-style interpolation supports a number of more advanced -syntaxes, too. - -@table @code - -@item propname:-replacement -If @code{propname} exists, substitute its value; otherwise, -substitute @code{replacement}. @code{replacement} may be empty -(@code{%(propname:-)s}) - -@item propname:+replacement -If @code{propname} exists, substitute @code{replacement}; otherwise, -substitute an empty string. - -@end table - -Although these are similar to shell substitutions, no other -substitutions are currently supported, and @code{replacement} in the -above cannot contain more substitutions. - -Note: like python, you can either do positional-argument interpolation -@emph{or} keyword-argument interpolation, not both. Thus you cannot use -a string like @code{WithProperties("foo-%(revision)s-%s", "branch")}. - -@heading Common Build Properties - -The following build properties are set when the build is started, and -are available to all steps. - -@table @code -@item branch - -This comes from the build's SourceStamp, and describes which branch is -being checked out. This will be @code{None} (which interpolates into -@code{WithProperties} as an empty string) if the build is on the -default branch, which is generally the trunk. Otherwise it will be a -string like ``branches/beta1.4''. The exact syntax depends upon the VC -system being used. - -@item revision - -This also comes from the SourceStamp, and is the revision of the source code -tree that was requested from the VC system. When a build is requested of a -specific revision (as is generally the case when the build is triggered by -Changes), this will contain the revision specification. This is always a -string, although the syntax depends upon the VC system in use: for SVN it is an -integer, for Mercurial it is a short string, for Darcs it is a rather large -string, etc. - -If the ``force build'' button was pressed, the revision will be @code{None}, -which means to use the most recent revision available. This is a ``trunk -build''. This will be interpolated as an empty string. - -@item got_revision - -This is set when a Source step checks out the source tree, and -provides the revision that was actually obtained from the VC system. -In general this should be the same as @code{revision}, except for -trunk builds, where @code{got_revision} indicates what revision was -current when the checkout was performed. This can be used to rebuild -the same source code later. - -Note that for some VC systems (Darcs in particular), the revision is a -large string containing newlines, and is not suitable for interpolation -into a filename. - -@item buildername - -This is a string that indicates which Builder the build was a part of. -The combination of buildername and buildnumber uniquely identify a -build. - -@item buildnumber - -Each build gets a number, scoped to the Builder (so the first build -performed on any given Builder will have a build number of 0). This -integer property contains the build's number. - -@item slavename - -This is a string which identifies which buildslave the build is -running on. - -@item scheduler - -If the build was started from a scheduler, then this property will -contain the name of that scheduler. - -@end table - - -@node Source Checkout, ShellCommand, Using Build Properties, Build Steps -@subsection Source Checkout - -The first step of any build is typically to acquire the source code -from which the build will be performed. There are several classes to -handle this, one for each of the different source control system that -Buildbot knows about. For a description of how Buildbot treats source -control in general, see @ref{Version Control Systems}. - -All source checkout steps accept some common parameters to control how -they get the sources and where they should be placed. The remaining -per-VC-system parameters are mostly to specify where exactly the -sources are coming from. - -@table @code -@item mode - -a string describing the kind of VC operation that is desired. Defaults -to @code{update}. - -@table @code -@item update -specifies that the CVS checkout/update should be performed directly -into the workdir. Each build is performed in the same directory, -allowing for incremental builds. This minimizes disk space, bandwidth, -and CPU time. However, it may encounter problems if the build process -does not handle dependencies properly (sometimes you must do a ``clean -build'' to make sure everything gets compiled), or if source files are -deleted but generated files can influence test behavior (e.g. python's -.pyc files), or when source directories are deleted but generated -files prevent CVS from removing them. Builds ought to be correct -regardless of whether they are done ``from scratch'' or incrementally, -but it is useful to test both kinds: this mode exercises the -incremental-build style. - -@item copy -specifies that the CVS workspace should be maintained in a separate -directory (called the 'copydir'), using checkout or update as -necessary. For each build, a new workdir is created with a copy of the -source tree (rm -rf workdir; cp -r copydir workdir). This doubles the -disk space required, but keeps the bandwidth low (update instead of a -full checkout). A full 'clean' build is performed each time. This -avoids any generated-file build problems, but is still occasionally -vulnerable to CVS problems such as a repository being manually -rearranged, causing CVS errors on update which are not an issue with a -full checkout. - -@c TODO: something is screwy about this, revisit. Is it the source -@c directory or the working directory that is deleted each time? - -@item clobber -specifes that the working directory should be deleted each time, -necessitating a full checkout for each build. This insures a clean -build off a complete checkout, avoiding any of the problems described -above. This mode exercises the ``from-scratch'' build style. - -@item export -this is like @code{clobber}, except that the 'cvs export' command is -used to create the working directory. This command removes all CVS -metadata files (the CVS/ directories) from the tree, which is -sometimes useful for creating source tarballs (to avoid including the -metadata in the tar file). -@end table - -@item workdir -like all Steps, this indicates the directory where the build will take -place. Source Steps are special in that they perform some operations -outside of the workdir (like creating the workdir itself). - -@item alwaysUseLatest -if True, bypass the usual ``update to the last Change'' behavior, and -always update to the latest changes instead. - -@item retry -If set, this specifies a tuple of @code{(delay, repeats)} which means -that when a full VC checkout fails, it should be retried up to -@var{repeats} times, waiting @var{delay} seconds between attempts. If -you don't provide this, it defaults to @code{None}, which means VC -operations should not be retried. This is provided to make life easier -for buildslaves which are stuck behind poor network connections. - -@end table - - -My habit as a developer is to do a @code{cvs update} and @code{make} each -morning. Problems can occur, either because of bad code being checked in, or -by incomplete dependencies causing a partial rebuild to fail where a -complete from-scratch build might succeed. A quick Builder which emulates -this incremental-build behavior would use the @code{mode='update'} -setting. - -On the other hand, other kinds of dependency problems can cause a clean -build to fail where a partial build might succeed. This frequently results -from a link step that depends upon an object file that was removed from a -later version of the tree: in the partial tree, the object file is still -around (even though the Makefiles no longer know how to create it). - -``official'' builds (traceable builds performed from a known set of -source revisions) are always done as clean builds, to make sure it is -not influenced by any uncontrolled factors (like leftover files from a -previous build). A ``full'' Builder which behaves this way would want -to use the @code{mode='clobber'} setting. - -Each VC system has a corresponding source checkout class: their -arguments are described on the following pages. - - -@menu -* CVS:: -* SVN:: -* Darcs:: -* Mercurial:: -* Arch:: -* Bazaar:: -* Bzr:: -* P4:: -* Git:: -* BitKeeper:: -@end menu - -@node CVS, SVN, Source Checkout, Source Checkout -@subsubsection CVS -@cindex CVS Checkout -@bsindex buildbot.steps.source.CVS - - -The @code{CVS} build step performs a @uref{http://www.nongnu.org/cvs/, -CVS} checkout or update. It takes the following arguments: - -@table @code -@item cvsroot -(required): specify the CVSROOT value, which points to a CVS -repository, probably on a remote machine. For example, the cvsroot -value you would use to get a copy of the Buildbot source code is -@code{:pserver:anonymous@@cvs.sourceforge.net:/cvsroot/buildbot} - -@item cvsmodule -(required): specify the cvs @code{module}, which is generally a -subdirectory of the CVSROOT. The cvsmodule for the Buildbot source -code is @code{buildbot}. - -@item branch -a string which will be used in a @code{-r} argument. This is most -useful for specifying a branch to work on. Defaults to @code{HEAD}. - -@item global_options -a list of flags to be put before the verb in the CVS command. - -@item checkoutDelay -if set, the number of seconds to put between the timestamp of the last -known Change and the value used for the @code{-D} option. Defaults to -half of the parent Build's treeStableTimer. - -@end table - - -@node SVN, Darcs, CVS, Source Checkout -@subsubsection SVN - -@cindex SVN Checkout -@bsindex buildbot.steps.source.SVN - - -The @code{SVN} build step performs a -@uref{http://subversion.tigris.org, Subversion} checkout or update. -There are two basic ways of setting up the checkout step, depending -upon whether you are using multiple branches or not. - -If all of your builds use the same branch, then you should create the -@code{SVN} step with the @code{svnurl} argument: - -@table @code -@item svnurl -(required): this specifies the @code{URL} argument that will be given -to the @code{svn checkout} command. It dictates both where the -repository is located and which sub-tree should be extracted. In this -respect, it is like a combination of the CVS @code{cvsroot} and -@code{cvsmodule} arguments. For example, if you are using a remote -Subversion repository which is accessible through HTTP at a URL of -@code{http://svn.example.com/repos}, and you wanted to check out the -@code{trunk/calc} sub-tree, you would use -@code{svnurl="http://svn.example.com/repos/trunk/calc"} as an argument -to your @code{SVN} step. -@end table - -If, on the other hand, you are building from multiple branches, then -you should create the @code{SVN} step with the @code{baseURL} and -@code{defaultBranch} arguments instead: - -@table @code -@item baseURL -(required): this specifies the base repository URL, to which a branch -name will be appended. It should probably end in a slash. - -@item defaultBranch -(optional): this specifies the name of the branch to use when a Build -does not provide one of its own. This will be appended to -@code{baseURL} to create the string that will be passed to the -@code{svn checkout} command. - -@item username -(optional): if specified, this will be passed to the @code{svn} -binary with a @code{--username} option. - -@item password -(optional): if specified, this will be passed to the @code{svn} -binary with a @code{--password} option. The password itself will be -suitably obfuscated in the logs. - -@item extra_args -(optional): if specified, an array of strings that will be passed as -extra arguments to the @code{svn} binary. - -@item keep_on_purge -(optional): specific files or directories to keep between purges, -like some build outputs that can be reused between builds. - -@item ignore_ignores -(optional): when purging changes, don't use rules defined in -svn:ignore properties and global-ignores in subversion/config. - -@item always_purge -(optional): if set to True, always purge local changes after each -build. This is everything that would appear in a @code{svn status}. - -@item depth -(optional): Specify depth argument to achieve sparse checkout. Only available if slave has Subversion 1.5 or higher. - -If set to "empty" updates will not pull in any files or subdirectories not already present. If set to "files", updates will pull in any files not already present, but not directories. If set to "immediates", updates willl pull in any files or subdirectories not already present, the new subdirectories will have depth: empty. If set to "infinity", updates will pull in any files or subdirectories not already present; the new subdirectories will have depth-infinity. Infinity is equivalent to SVN default update behavior, without specifying any depth argument. - -@end table - -If you are using branches, you must also make sure your -@code{ChangeSource} will report the correct branch names. - -@heading branch example - -Let's suppose that the ``MyProject'' repository uses branches for the -trunk, for various users' individual development efforts, and for -several new features that will require some amount of work (involving -multiple developers) before they are ready to merge onto the trunk. -Such a repository might be organized as follows: - -@example -svn://svn.example.org/MyProject/trunk -svn://svn.example.org/MyProject/branches/User1/foo -svn://svn.example.org/MyProject/branches/User1/bar -svn://svn.example.org/MyProject/branches/User2/baz -svn://svn.example.org/MyProject/features/newthing -svn://svn.example.org/MyProject/features/otherthing -@end example - -Further assume that we want the Buildbot to run tests against the -trunk and against all the feature branches (i.e., do a -checkout/compile/build of branch X when a file has been changed on -branch X, when X is in the set [trunk, features/newthing, -features/otherthing]). We do not want the Buildbot to automatically -build any of the user branches, but it should be willing to build a -user branch when explicitly requested (most likely by the user who -owns that branch). - -There are three things that need to be set up to accomodate this -system. The first is a ChangeSource that is capable of identifying the -branch which owns any given file. This depends upon a user-supplied -function, in an external program that runs in the SVN commit hook and -connects to the buildmaster's @code{PBChangeSource} over a TCP -connection. (you can use the ``@code{buildbot sendchange}'' utility -for this purpose, but you will still need an external program to -decide what value should be passed to the @code{--branch=} argument). -For example, a change to a file with the SVN url of -``svn://svn.example.org/MyProject/features/newthing/src/foo.c'' should -be broken down into a Change instance with -@code{branch='features/newthing'} and @code{file='src/foo.c'}. - -The second piece is an @code{AnyBranchScheduler} which will pay -attention to the desired branches. It will not pay attention to the -user branches, so it will not automatically start builds in response -to changes there. The AnyBranchScheduler class requires you to -explicitly list all the branches you want it to use, but it would not -be difficult to write a subclass which used -@code{branch.startswith('features/'} to remove the need for this -explicit list. Or, if you want to build user branches too, you can use -AnyBranchScheduler with @code{branches=None} to indicate that you want -it to pay attention to all branches. - -The third piece is an @code{SVN} checkout step that is configured to -handle the branches correctly, with a @code{baseURL} value that -matches the way the ChangeSource splits each file's URL into base, -branch, and file. - -@example -from buildbot.changes.pb import PBChangeSource -from buildbot.scheduler import AnyBranchScheduler -from buildbot.process import source, factory -from buildbot.steps import source, shell - -c['change_source'] = PBChangeSource() -s1 = AnyBranchScheduler('main', - ['trunk', 'features/newthing', 'features/otherthing'], - 10*60, ['test-i386', 'test-ppc']) -c['schedulers'] = [s1] - -f = factory.BuildFactory() -f.addStep(source.SVN(mode='update', - baseURL='svn://svn.example.org/MyProject/', - defaultBranch='trunk')) -f.addStep(shell.Compile(command="make all")) -f.addStep(shell.Test(command="make test")) - -c['builders'] = [ - @{'name':'test-i386', 'slavename':'bot-i386', 'builddir':'test-i386', - 'factory':f @}, - @{'name':'test-ppc', 'slavename':'bot-ppc', 'builddir':'test-ppc', - 'factory':f @}, - ] -@end example - -In this example, when a change arrives with a @code{branch} attribute -of ``trunk'', the resulting build will have an SVN step that -concatenates ``svn://svn.example.org/MyProject/'' (the baseURL) with -``trunk'' (the branch name) to get the correct svn command. If the -``newthing'' branch has a change to ``src/foo.c'', then the SVN step -will concatenate ``svn://svn.example.org/MyProject/'' with -``features/newthing'' to get the svnurl for checkout. - -@node Darcs, Mercurial, SVN, Source Checkout -@subsubsection Darcs - -@cindex Darcs Checkout -@bsindex buildbot.steps.source.Darcs - - -The @code{Darcs} build step performs a -@uref{http://darcs.net/, Darcs} checkout or update. - -Like @xref{SVN}, this step can either be configured to always check -out a specific tree, or set up to pull from a particular branch that -gets specified separately for each build. Also like SVN, the -repository URL given to Darcs is created by concatenating a -@code{baseURL} with the branch name, and if no particular branch is -requested, it uses a @code{defaultBranch}. The only difference in -usage is that each potential Darcs repository URL must point to a -fully-fledged repository, whereas SVN URLs usually point to sub-trees -of the main Subversion repository. In other words, doing an SVN -checkout of @code{baseURL} is legal, but silly, since you'd probably -wind up with a copy of every single branch in the whole repository. -Doing a Darcs checkout of @code{baseURL} is just plain wrong, since -the parent directory of a collection of Darcs repositories is not -itself a valid repository. - -The Darcs step takes the following arguments: - -@table @code -@item repourl -(required unless @code{baseURL} is provided): the URL at which the -Darcs source repository is available. - -@item baseURL -(required unless @code{repourl} is provided): the base repository URL, -to which a branch name will be appended. It should probably end in a -slash. - -@item defaultBranch -(allowed if and only if @code{baseURL} is provided): this specifies -the name of the branch to use when a Build does not provide one of its -own. This will be appended to @code{baseURL} to create the string that -will be passed to the @code{darcs get} command. -@end table - -@node Mercurial, Arch, Darcs, Source Checkout -@subsubsection Mercurial - -@cindex Mercurial Checkout -@bsindex buildbot.steps.source.Mercurial - - -The @code{Mercurial} build step performs a -@uref{http://selenic.com/mercurial, Mercurial} (aka ``hg'') checkout -or update. - -Branches are available in two modes: ''dirname'' like @xref{Darcs}, or -''inrepo'', which uses the repository internal branches. Make sure this -setting matches your changehook, if you have that installed. - -The Mercurial step takes the following arguments: - -@table @code -@item repourl -(required unless @code{baseURL} is provided): the URL at which the -Mercurial source repository is available. - -@item baseURL -(required unless @code{repourl} is provided): the base repository URL, -to which a branch name will be appended. It should probably end in a -slash. - -@item defaultBranch -(allowed if and only if @code{baseURL} is provided): this specifies -the name of the branch to use when a Build does not provide one of its -own. This will be appended to @code{baseURL} to create the string that -will be passed to the @code{hg clone} command. - -@item branchType -either 'dirname' (default) or 'inrepo' depending on whether -the branch name should be appended to the @code{baseURL} -or the branch is a mercurial named branch and can be -found within the @code{repourl}. - -@item clobberOnBranchChange -boolean, defaults to True. If set and -using inrepos branches, clobber the tree -at each branch change. Otherwise, just -update to the branch. - -@end table - - -@node Arch, Bazaar, Mercurial, Source Checkout -@subsubsection Arch - -@cindex Arch Checkout -@bsindex buildbot.steps.source.Arch - - -The @code{Arch} build step performs an @uref{http://gnuarch.org/, -Arch} checkout or update using the @code{tla} client. It takes the -following arguments: - -@table @code -@item url -(required): this specifies the URL at which the Arch source archive is -available. - -@item version -(required): this specifies which ``development line'' (like a branch) -should be used. This provides the default branch name, but individual -builds may specify a different one. - -@item archive -(optional): Each repository knows its own archive name. If this -parameter is provided, it must match the repository's archive name. -The parameter is accepted for compatibility with the @code{Bazaar} -step, below. - -@end table - -@node Bazaar, Bzr, Arch, Source Checkout -@subsubsection Bazaar - -@cindex Bazaar Checkout -@bsindex buildbot.steps.source.Bazaar - - -@code{Bazaar} is an alternate implementation of the Arch VC system, -which uses a client named @code{baz}. The checkout semantics are just -different enough from @code{tla} that there is a separate BuildStep for -it. - -It takes exactly the same arguments as @code{Arch}, except that the -@code{archive=} parameter is required. (baz does not emit the archive -name when you do @code{baz register-archive}, so we must provide it -ourselves). - - -@node Bzr, P4, Bazaar, Source Checkout -@subsubsection Bzr - -@cindex Bzr Checkout -@bsindex buildbot.steps.source.Bzr - -@code{bzr} is a descendant of Arch/Baz, and is frequently referred to -as simply ``Bazaar''. The repository-vs-workspace model is similar to -Darcs, but it uses a strictly linear sequence of revisions (one -history per branch) like Arch. Branches are put in subdirectories. -This makes it look very much like Mercurial. It takes the following -arguments: - -@table @code - -@item repourl -(required unless @code{baseURL} is provided): the URL at which the -Bzr source repository is available. - -@item baseURL -(required unless @code{repourl} is provided): the base repository URL, -to which a branch name will be appended. It should probably end in a -slash. - -@item defaultBranch -(allowed if and only if @code{baseURL} is provided): this specifies -the name of the branch to use when a Build does not provide one of its -own. This will be appended to @code{baseURL} to create the string that -will be passed to the @code{bzr checkout} command. - -@item forceSharedRepo -(boolean, optional, defaults to False): If set to True, the working directory -will be made into a bzr shared repository if it is not already. Shared -repository greatly reduces the amount of history data that needs to be -downloaded if not using update/copy mode, or if using update/copy mode with -multiple branches. -@end table - - - -@node P4, Git, Bzr, Source Checkout -@subsubsection P4 - -@cindex Perforce Update -@bsindex buildbot.steps.source.P4 -@c TODO @bsindex buildbot.steps.source.P4Sync - - -The @code{P4} build step creates a @uref{http://www.perforce.com/, -Perforce} client specification and performs an update. - -@table @code -@item p4base -A view into the Perforce depot without branch name or trailing "...". -Typically "//depot/proj/". -@item defaultBranch -A branch name to append on build requests if none is specified. -Typically "trunk". -@item p4port -(optional): the host:port string describing how to get to the P4 Depot -(repository), used as the -p argument for all p4 commands. -@item p4user -(optional): the Perforce user, used as the -u argument to all p4 -commands. -@item p4passwd -(optional): the Perforce password, used as the -p argument to all p4 -commands. -@item p4extra_views -(optional): a list of (depotpath, clientpath) tuples containing extra -views to be mapped into the client specification. Both will have -"/..." appended automatically. The client name and source directory -will be prepended to the client path. -@item p4client -(optional): The name of the client to use. In mode='copy' and -mode='update', it's particularly important that a unique name is used -for each checkout directory to avoid incorrect synchronization. For -this reason, Python percent substitution will be performed on this value -to replace %(slave)s with the slave name and %(builder)s with the -builder name. The default is "buildbot_%(slave)s_%(build)s". -@end table - - -@node Git, BitKeeper, P4, Source Checkout -@subsubsection Git - -@cindex Git Checkout -@bsindex buildbot.steps.source.Git - -The @code{Git} build step clones or updates a @uref{http://git.or.cz/, -Git} repository and checks out the specified branch or revision. Note -that the buildbot supports Git version 1.2.0 and later: earlier -versions (such as the one shipped in Ubuntu 'Dapper') do not support -the @command{git init} command that the buildbot uses. - -The Git step takes the following arguments: - -@table @code -@item repourl -(required): the URL of the upstream Git repository. - -@item branch -(optional): this specifies the name of the branch to use when a Build -does not provide one of its own. If this this parameter is not -specified, and the Build does not provide a branch, the ``master'' -branch will be used. - -@item ignore_ignores -(optional): when purging changes, don't use .gitignore and -.git/info/exclude. - -@end table - - -@node BitKeeper, , Git, Source Checkout -@subsubsection BitKeeper - -@cindex BitKeeper Checkout -@bsindex buildbot.steps.source.BK - -The @code{BK} build step performs a @uref{http://www.bitkeeper.com/, BitKeeper} -checkout or update. - -The BitKeeper step takes the following arguments: - -@table @code -@item repourl -(required unless @code{baseURL} is provided): the URL at which the -BitKeeper source repository is available. - -@item baseURL -(required unless @code{repourl} is provided): the base repository URL, -to which a branch name will be appended. It should probably end in a -slash. - -@end table - - -@node ShellCommand, Simple ShellCommand Subclasses, Source Checkout, Build Steps -@subsection ShellCommand - -@bsindex buildbot.steps.shell.ShellCommand -@c TODO @bsindex buildbot.steps.shell.TreeSize - -This is a useful base class for just about everything you might want -to do during a build (except for the initial source checkout). It runs -a single command in a child shell on the buildslave. All stdout/stderr -is recorded into a LogFile. The step finishes with a status of FAILURE -if the command's exit code is non-zero, otherwise it has a status of -SUCCESS. - -The preferred way to specify the command is with a list of argv strings, -since this allows for spaces in filenames and avoids doing any fragile -shell-escaping. You can also specify the command with a single string, in -which case the string is given to '/bin/sh -c COMMAND' for parsing. - -On Windows, commands are run via @code{cmd.exe /c} which works well. However, -if you're running a batch file, the error level does not get propagated -correctly unless you add 'call' before your batch file's name: -@code{cmd=['call', 'myfile.bat', ...]}. - -@code{ShellCommand} arguments: - -@table @code -@item command -a list of strings (preferred) or single string (discouraged) which -specifies the command to be run. A list of strings is preferred -because it can be used directly as an argv array. Using a single -string (with embedded spaces) requires the buildslave to pass the -string to /bin/sh for interpretation, which raises all sorts of -difficult questions about how to escape or interpret shell -metacharacters. - -@item workdir -All ShellCommands are run by default in the ``workdir'', which -defaults to the ``@file{build}'' subdirectory of the slave builder's -base directory. The absolute path of the workdir will thus be the -slave's basedir (set as an option to @code{buildbot create-slave}, -@pxref{Creating a buildslave}) plus the builder's basedir (set in the -builder's @code{c['builddir']} key in master.cfg) plus the workdir -itself (a class-level attribute of the BuildFactory, defaults to -``@file{build}''). - -For example: - -@example -f.addStep(ShellCommand(command=["make", "test"], - workdir="build/tests")) -@end example - -@item env -a dictionary of environment strings which will be added to the child -command's environment. For example, to run tests with a different i18n -language setting, you might use - -@example -f.addStep(ShellCommand(command=["make", "test"], - env=@{'LANG': 'fr_FR'@})) -@end example - -These variable settings will override any existing ones in the -buildslave's environment or the environment specified in the -Builder. The exception is PYTHONPATH, which is merged -with (actually prepended to) any existing $PYTHONPATH setting. The -value is treated as a list of directories to prepend, and a single -string is treated like a one-item list. For example, to prepend both -@file{/usr/local/lib/python2.3} and @file{/home/buildbot/lib/python} -to any existing $PYTHONPATH setting, you would do something like the -following: - -@example -f.addStep(ShellCommand( - command=["make", "test"], - env=@{'PYTHONPATH': ["/usr/local/lib/python2.3", - "/home/buildbot/lib/python"] @})) -@end example - -@item want_stdout -if False, stdout from the child process is discarded rather than being -sent to the buildmaster for inclusion in the step's LogFile. - -@item want_stderr -like @code{want_stdout} but for stderr. Note that commands run through -a PTY do not have separate stdout/stderr streams: both are merged into -stdout. - -@item usePTY -Should this command be run in a @code{pty}? The default is to observe the -configuration of the client (@pxref{Buildslave Options}), but specifying -@code{True} or @code{False} here will override the default. - -The advantage of using a PTY is that ``grandchild'' processes are more likely -to be cleaned up if the build is interrupted or times out (since it enables the -use of a ``process group'' in which all child processes will be placed). The -disadvantages: some forms of Unix have problems with PTYs, some of your unit -tests may behave differently when run under a PTY (generally those which check -to see if they are being run interactively), and PTYs will merge the stdout and -stderr streams into a single output stream (which means the red-vs-black -coloring in the logfiles will be lost). - -@item logfiles -Sometimes commands will log interesting data to a local file, rather -than emitting everything to stdout or stderr. For example, Twisted's -``trial'' command (which runs unit tests) only presents summary -information to stdout, and puts the rest into a file named -@file{_trial_temp/test.log}. It is often useful to watch these files -as the command runs, rather than using @command{/bin/cat} to dump -their contents afterwards. - -The @code{logfiles=} argument allows you to collect data from these -secondary logfiles in near-real-time, as the step is running. It -accepts a dictionary which maps from a local Log name (which is how -the log data is presented in the build results) to either a remote filename -(interpreted relative to the build's working directory), or a dictionary -of options. Each named file will be polled on a regular basis (every couple -of seconds) as the build runs, and any new text will be sent over to the -buildmaster. - -If you provide a dictionary of options instead of a string, you must specify -the @code{filename} key. You can optionally provide a @code{follow} key which -is a boolean controlling whether a logfile is followed or concatenated in its -entirety. Following is appropriate for logfiles to which the build step will -append, where the pre-existing contents are not interesting. The default value -for @code{follow} is @code{False}, which gives the same behavior as just -providing a string filename. - -@example -f.addStep(ShellCommand( - command=["make", "test"], - logfiles=@{"triallog": "_trial_temp/test.log"@})) -@end example - -@example -f.addStep(ShellCommand( - command=["make", "test"], - logfiles=@{"triallog": @{"filename": "_trial_temp/test.log", - "follow": True,@}@})) -@end example - - -@item lazylogfiles -If set to @code{True}, logfiles will be tracked lazily, meaning that they will -only be added when and if something is written to them. This can be used to -suppress the display of empty or missing log files. The default is @code{False}. - - -@item timeout -if the command fails to produce any output for this many seconds, it -is assumed to be locked up and will be killed. - -@item maxTime -if the command takes longer than this many seconds, it will be killed. - -@item description -This will be used to describe the command (on the Waterfall display) -while the command is still running. It should be a single -imperfect-tense verb, like ``compiling'' or ``testing''. The preferred -form is a list of short strings, which allows the HTML -displays to create narrower columns by emitting a
tag between each -word. You may also provide a single string. - -@item descriptionDone -This will be used to describe the command once it has finished. A -simple noun like ``compile'' or ``tests'' should be used. Like -@code{description}, this may either be a list of short strings or a -single string. - -If neither @code{description} nor @code{descriptionDone} are set, the -actual command arguments will be used to construct the description. -This may be a bit too wide to fit comfortably on the Waterfall -display. - -@example -f.addStep(ShellCommand(command=["make", "test"], - description=["testing"], - descriptionDone=["tests"])) -@end example - -@item logEnviron -If this option is true (the default), then the step's logfile will describe the -environment variables on the slave. In situations where the environment is not -relevant and is long, it may be easier to set @code{logEnviron=False}. - -@end table - -@node Simple ShellCommand Subclasses, Python BuildSteps, ShellCommand, Build Steps -@subsection Simple ShellCommand Subclasses - -Several subclasses of ShellCommand are provided as starting points for -common build steps. These are all very simple: they just override a few -parameters so you don't have to specify them yourself, making the master.cfg -file less verbose. - -@menu -* Configure:: -* Compile:: -* Test:: -* TreeSize:: -* PerlModuleTest:: -* Testing with mysql-test-run:: -* SetProperty:: -* SubunitShellCommand:: -@end menu - -@node Configure, Compile, Simple ShellCommand Subclasses, Simple ShellCommand Subclasses -@subsubsection Configure - -@bsindex buildbot.steps.shell.Configure - -This is intended to handle the @code{./configure} step from -autoconf-style projects, or the @code{perl Makefile.PL} step from perl -MakeMaker.pm-style modules. The default command is @code{./configure} -but you can change this by providing a @code{command=} parameter. - -@node Compile, Test, Configure, Simple ShellCommand Subclasses -@subsubsection Compile - -@bsindex buildbot.steps.shell.Compile - -This is meant to handle compiling or building a project written in C. -The default command is @code{make all}. When the compile is finished, -the log file is scanned for GCC warning messages, a summary log is -created with any problems that were seen, and the step is marked as -WARNINGS if any were discovered. Through the @code{WarningCountingShellCommand} -superclass, the number of warnings is stored in a Build Property named -``warnings-count'', which is accumulated over all Compile steps (so if two -warnings are found in one step, and three are found in another step, the -overall build will have a ``warnings-count'' property of 5). - -The default regular expression used to detect a warning is -@code{'.*warning[: ].*'} , which is fairly liberal and may cause -false-positives. To use a different regexp, provide a -@code{warningPattern=} argument, or use a subclass which sets the -@code{warningPattern} attribute: - -@example -f.addStep(Compile(command=["make", "test"], - warningPattern="^Warning: ")) -@end example - -The @code{warningPattern=} can also be a pre-compiled python regexp -object: this makes it possible to add flags like @code{re.I} (to use -case-insensitive matching). - -Note that the compiled @code{warningPattern} will have its @code{match} method -called, which is subtly different from a @code{search}. Your regular -expression must match the from the beginning of the line. This means that to -look for the word "warning" in the middle of a line, you will need to -prepend @code{'.*'} to your regular expression. - -The @code{suppressionFile=} argument can be specified as the (relative) path -of a file inside the workdir defining warnings to be suppressed from the -warning counting and log file. The file will be uploaded to the master from -the slave before compiling, and any warning matched by a line in the -suppression file will be ignored. This is useful to accept certain warnings -(eg. in some special module of the source tree or in cases where the compiler -is being particularly stupid), yet still be able to easily detect and fix the -introduction of new warnings. - -The file must contain one line per pattern of warnings to ignore. Empty lines -and lines beginning with @code{#} are ignored. Other lines must consist of a -regexp matching the file name, followed by a colon (@code{:}), followed by a -regexp matching the text of the warning. Optionally this may be followed by -another colon and a line number range. For example: - -@example -# Sample warning suppression file - -mi_packrec.c : .*result of 32-bit shift implicitly converted to 64 bits.* : 560-600 -DictTabInfo.cpp : .*invalid access to non-static.* -kernel_types.h : .*only defines private constructors and has no friends.* : 51 -@end example - -If no line number range is specified, the pattern matches the whole file; if -only one number is given it matches only on that line. - -The default warningPattern regexp only matches the warning text, so line -numbers and file names are ignored. To enable line number and file name -matching, privide a different regexp and provide a function (callable) as the -argument of @code{warningExtractor=}. The function is called with three -arguments: the BuildStep object, the line in the log file with the warning, -and the @code{SRE_Match} object of the regexp search for @code{warningPattern}. It -should return a tuple @code{(filename, linenumber, warning_test)}. For -example: - -@example -f.addStep(Compile(command=["make"], - warningPattern="^(.*?):([0-9]+): [Ww]arning: (.*)$", - warningExtractor=Compile.warnExtractFromRegexpGroups, - suppressionFile="support-files/compiler_warnings.supp")) -@end example - -(@code{Compile.warnExtractFromRegexpGroups} is a pre-defined function that -returns the filename, linenumber, and text from groups (1,2,3) of the regexp -match). - -In projects with source files in multiple directories, it is possible to get -full path names for file names matched in the suppression file, as long as the -build command outputs the names of directories as they are entered into and -left again. For this, specify regexps for the arguments -@code{directoryEnterPattern=} and @code{directoryLeavePattern=}. The -@code{directoryEnterPattern=} regexp should return the name of the directory -entered into in the first matched group. The defaults, which are suitable for -GNU Make, are these: - -@example -directoryEnterPattern = "make.*: Entering directory [\"`'](.*)['`\"]" -directoryLeavePattern = "make.*: Leaving directory" -@end example - -(TODO: this step needs to be extended to look for GCC error messages -as well, and collect them into a separate logfile, along with the -source code filenames involved). - - -@node Test, TreeSize, Compile, Simple ShellCommand Subclasses -@subsubsection Test - -@bsindex buildbot.steps.shell.Test - -This is meant to handle unit tests. The default command is @code{make -test}, and the @code{warnOnFailure} flag is set. - -@node TreeSize, PerlModuleTest, Test, Simple ShellCommand Subclasses -@subsubsection TreeSize - -@bsindex buildbot.steps.shell.TreeSize - -This is a simple command that uses the 'du' tool to measure the size -of the code tree. It puts the size (as a count of 1024-byte blocks, -aka 'KiB' or 'kibibytes') on the step's status text, and sets a build -property named 'tree-size-KiB' with the same value. - -@node PerlModuleTest, Testing with mysql-test-run, TreeSize, Simple ShellCommand Subclasses -@subsubsection PerlModuleTest - -@bsindex buildbot.steps.shell.PerlModuleTest - -This is a simple command that knows how to run tests of perl modules. -It parses the output to determine the number of tests passed and -failed and total number executed, saving the results for later query. - -@node Testing with mysql-test-run, SetProperty, PerlModuleTest, Simple ShellCommand Subclasses -@subsection Testing with mysql-test-run - -The @code{process.mtrlogobserver.MTR} class is a subclass of @code{Test} -(@ref{Test}). It is used to run test suites using the mysql-test-run program, -as used in MySQL, Drizzle, MariaDB, and MySQL storage engine plugins. - -The shell command to run the test suite is specified in the same way as for -the Test class. The MTR class will parse the output of running the test suite, -and use the count of tests executed so far to provide more accurate completion -time estimates. Any test failures that occur during the test are summarized on -the Waterfall Display. - -Server error logs are added as additional log files, useful to debug test -failures. - -Optionally, data about the test run and any test failures can be inserted into -a database for further analysis and report generation. To use this facility, -create an instance of @code{twisted.enterprise.adbapi.ConnectionPool} with -connections to the database. The necessary tables can be created automatically -by setting @code{autoCreateTables} to @code{True}, or manually using the SQL -found in the @file{mtrlogobserver.py} source file. - -One problem with specifying a database is that each reload of the -configuration will get a new instance of @code{ConnectionPool} (even if the -connection parameters are the same). To avoid that Buildbot thinks the builder -configuration has changed because of this, use the -@code{process.mtrlogobserver.EqConnectionPool} subclass of -@code{ConnectionPool}, which implements an equiality operation that avoids -this problem. - -Example use: - -@example -from buildbot.process.mtrlogobserver import MTR, EqConnectionPool -myPool = EqConnectionPool("MySQLdb", "host", "buildbot", "password", "db") -myFactory.addStep(MTR(workdir="mysql-test", dbpool=myPool, - command=["perl", "mysql-test-run.pl", "--force"])) -@end example - -@code{MTR} arguments: - -@table @code - -@item textLimit -Maximum number of test failures to show on the waterfall page (to not flood -the page in case of a large number of test failures. Defaults to 5. - -@item testNameLimit -Maximum length of test names to show unabbreviated in the waterfall page, to -avoid excessive column width. Defaults to 16. - -@item parallel -Value of @code{--parallel} option used for mysql-test-run.pl (number of processes -used to run the test suite in parallel). Defaults to 4. This is used to -determine the number of server error log files to download from the -slave. Specifying a too high value does not hurt (as nonexisting error logs -will be ignored), however if using @code{--parallel} value greater than the default -it needs to be specified, or some server error logs will be missing. - -@item dbpool -An instance of twisted.enterprise.adbapi.ConnectionPool, or None. Defaults to -None. If specified, results are inserted into the database using the -ConnectionPool. - -@item autoCreateTables -Boolean, defaults to False. If True (and @code{dbpool} is specified), the -necessary database tables will be created automatically if they do not exist -already. Alternatively, the tables can be created manually from the SQL -statements found in the mtrlogobserver.py source file. - -@item test_type -Short string that will be inserted into the database in the row for the test -run. Defaults to the empty string, but can be specified to identify different -types of test runs. - -@item test_info -Descriptive string that will be inserted into the database in the row for the test -run. Defaults to the empty string, but can be specified as a user-readable -description of this particular test run. - -@item mtr_subdir -The subdirectory in which to look for server error log files. Defaults to -``mysql-test'', which is usually correct. WithProperties is supported. - -@end table - - -@node SetProperty, SubunitShellCommand , Testing with mysql-test-run, Simple ShellCommand Subclasses -@subsubsection SetProperty - -@bsindex buildbot.steps.shell.SetProperty - -This buildstep is similar to ShellCommand, except that it captures the -output of the command into a property. It is usually used like this: - -@example -f.addStep(SetProperty(command="uname -a", property="uname")) -@end example - -This runs @code{uname -a} and captures its stdout, stripped of leading -and trailing whitespace, in the property "uname". To avoid stripping, -add @code{strip=False}. The @code{property} argument can be specified -as a @code{WithProperties} object. - -The more advanced usage allows you to specify a function to extract -properties from the command output. Here you can use regular -expressions, string interpolation, or whatever you would like. -The function is called with three arguments: the exit status of the -command, its standard output as a string, and its standard error as -a string. It should return a dictionary containing all new properties. - -@example -def glob2list(rc, stdout, stderr): - jpgs = [ l.strip() for l in stdout.split('\n') ] - return @{ 'jpgs' : jpgs @} -f.addStep(SetProperty(command="ls -1 *.jpg", extract_fn=glob2list)) -@end example - -Note that any ordering relationship of the contents of stdout and -stderr is lost. For example, given - -@example -f.addStep(SetProperty( - command="echo output1; echo error >&2; echo output2", - extract_fn=my_extract)) -@end example - -Then @code{my_extract} will see @code{stdout="output1\noutput2\n"} -and @code{stderr="error\n"}. - -@node SubunitShellCommand, , SetProperty, Simple ShellCommand Subclasses -@subsubsection SubunitShellCommand - -@bsindex buildbot.process.subunitlogger.SubunitShellCommand - -This buildstep is similar to ShellCommand, except that it runs the log content -through a subunit filter to extract test and failure counts. - -@example -f.addStep(SubunitShellCommand(command="make test")) -@end example - -This runs @code{make test} and filters it through subunit. The 'tests' and -'test failed' progress metrics will now accumulate test data from the test run. - -@node Python BuildSteps, Transferring Files, Simple ShellCommand Subclasses, Build Steps -@subsection Python BuildSteps - -Here are some BuildSteps that are specifcally useful for projects -implemented in Python. - -@menu -* BuildEPYDoc:: -* PyFlakes:: -* PyLint:: -@end menu - -@node BuildEPYDoc -@subsubsection BuildEPYDoc - -@bsindex buildbot.steps.python.BuildEPYDoc - -@url{http://epydoc.sourceforge.net/, epydoc} is a tool for generating -API documentation for Python modules from their docstrings. It reads -all the .py files from your source tree, processes the docstrings -therein, and creates a large tree of .html files (or a single .pdf -file). - -The @code{buildbot.steps.python.BuildEPYDoc} step will run -@command{epydoc} to produce this API documentation, and will count the -errors and warnings from its output. - -You must supply the command line to be used. The default is -@command{make epydocs}, which assumes that your project has a Makefile -with an ``epydocs'' target. You might wish to use something like -@command{epydoc -o apiref source/PKGNAME} instead. You might also want -to add @command{--pdf} to generate a PDF file instead of a large tree -of HTML files. - -The API docs are generated in-place in the build tree (under the -workdir, in the subdirectory controlled by the ``-o'' argument). To -make them useful, you will probably have to copy them to somewhere -they can be read. A command like @command{rsync -ad apiref/ -dev.example.com:~public_html/current-apiref/} might be useful. You -might instead want to bundle them into a tarball and publish it in the -same place where the generated install tarball is placed. - -@example -from buildbot.steps.python import BuildEPYDoc - -... -f.addStep(BuildEPYDoc(command=["epydoc", "-o", "apiref", "source/mypkg"])) -@end example - - -@node PyFlakes -@subsubsection PyFlakes - -@bsindex buildbot.steps.python.PyFlakes - -@url{http://divmod.org/trac/wiki/DivmodPyflakes, PyFlakes} is a tool -to perform basic static analysis of Python code to look for simple -errors, like missing imports and references of undefined names. It is -like a fast and simple form of the C ``lint'' program. Other tools -(like pychecker) provide more detailed results but take longer to run. - -The @code{buildbot.steps.python.PyFlakes} step will run pyflakes and -count the various kinds of errors and warnings it detects. - -You must supply the command line to be used. The default is -@command{make pyflakes}, which assumes you have a top-level Makefile -with a ``pyflakes'' target. You might want to use something like -@command{pyflakes .} or @command{pyflakes src}. - -@example -from buildbot.steps.python import PyFlakes - -... -f.addStep(PyFlakes(command=["pyflakes", "src"])) -@end example - -@node PyLint -@subsubsection PyLint - -@bsindex buildbot.steps.python.PyLint - -Similarly, the @code{buildbot.steps.python.PyLint} step will run pylint and -analyze the results. - -You must supply the command line to be used. There is no default. - -@example -from buildbot.steps.python import PyLint - -... -f.addStep(PyLint(command=["pylint", "src"])) -@end example - - -@node Transferring Files -@subsection Transferring Files - -@cindex File Transfer -@bsindex buildbot.steps.transfer.FileUpload -@bsindex buildbot.steps.transfer.FileDownload -@bsindex buildbot.steps.transfer.DirectoryUpload - -Most of the work involved in a build will take place on the -buildslave. But occasionally it is useful to do some work on the -buildmaster side. The most basic way to involve the buildmaster is -simply to move a file from the slave to the master, or vice versa. -There are a pair of BuildSteps named @code{FileUpload} and -@code{FileDownload} to provide this functionality. @code{FileUpload} -moves a file @emph{up to} the master, while @code{FileDownload} moves -a file @emph{down from} the master. - -As an example, let's assume that there is a step which produces an -HTML file within the source tree that contains some sort of generated -project documentation. We want to move this file to the buildmaster, -into a @file{~/public_html} directory, so it can be visible to -developers. This file will wind up in the slave-side working directory -under the name @file{docs/reference.html}. We want to put it into the -master-side @file{~/public_html/ref.html}. - -@example -from buildbot.steps.shell import ShellCommand -from buildbot.steps.transfer import FileUpload - -f.addStep(ShellCommand(command=["make", "docs"])) -f.addStep(FileUpload(slavesrc="docs/reference.html", - masterdest="~/public_html/ref.html")) -@end example - -The @code{masterdest=} argument will be passed to os.path.expanduser, -so things like ``~'' will be expanded properly. Non-absolute paths -will be interpreted relative to the buildmaster's base directory. -Likewise, the @code{slavesrc=} argument will be expanded and -interpreted relative to the builder's working directory. - - -To move a file from the master to the slave, use the -@code{FileDownload} command. For example, let's assume that some step -requires a configuration file that, for whatever reason, could not be -recorded in the source code repository or generated on the buildslave -side: - -@example -from buildbot.steps.shell import ShellCommand -from buildbot.steps.transfer import FileDownload - -f.addStep(FileDownload(mastersrc="~/todays_build_config.txt", - slavedest="build_config.txt")) -f.addStep(ShellCommand(command=["make", "config"])) -@end example - -Like @code{FileUpload}, the @code{mastersrc=} argument is interpreted -relative to the buildmaster's base directory, and the -@code{slavedest=} argument is relative to the builder's working -directory. If the buildslave is running in @file{~buildslave}, and the -builder's ``builddir'' is something like @file{tests-i386}, then the -workdir is going to be @file{~buildslave/tests-i386/build}, and a -@code{slavedest=} of @file{foo/bar.html} will get put in -@file{~buildslave/tests-i386/build/foo/bar.html}. Both of these commands -will create any missing intervening directories. - -@subheading Other Parameters - -The @code{maxsize=} argument lets you set a maximum size for the file -to be transferred. This may help to avoid surprises: transferring a -100MB coredump when you were expecting to move a 10kB status file -might take an awfully long time. The @code{blocksize=} argument -controls how the file is sent over the network: larger blocksizes are -slightly more efficient but also consume more memory on each end, and -there is a hard-coded limit of about 640kB. - -The @code{mode=} argument allows you to control the access permissions -of the target file, traditionally expressed as an octal integer. The -most common value is probably 0755, which sets the ``x'' executable -bit on the file (useful for shell scripts and the like). The default -value for @code{mode=} is None, which means the permission bits will -default to whatever the umask of the writing process is. The default -umask tends to be fairly restrictive, but at least on the buildslave -you can make it less restrictive with a --umask command-line option at -creation time (@pxref{Buildslave Options}). - -@subheading Transfering Directories - -To transfer complete directories from the buildslave to the master, there -is a BuildStep named @code{DirectoryUpload}. It works like @code{FileUpload}, -just for directories. However it does not support the @code{maxsize}, -@code{blocksize} and @code{mode} arguments. As an example, let's assume an -generated project documentation, which consists of many files (like the output -of doxygen or epydoc). We want to move the entire documentation to the -buildmaster, into a @code{~/public_html/docs} directory. On the slave-side -the directory can be found under @code{docs}: - -@example -from buildbot.steps.shell import ShellCommand -from buildbot.steps.transfer import DirectoryUpload - -f.addStep(ShellCommand(command=["make", "docs"])) -f.addStep(DirectoryUpload(slavesrc="docs", - masterdest="~/public_html/docs")) -@end example - -The DirectoryUpload step will create all necessary directories and -transfers empty directories, too. - -@node Steps That Run on the Master -@subsection Steps That Run on the Master - -Occasionally, it is useful to execute some task on the master, for example to -create a directory, deploy a build result, or trigger some other centralized -processing. This is possible, in a limited fashion, with the -@code{MasterShellCommand} step. - -This step operates similarly to a regular @code{ShellCommand}, but executes on -the master, instead of the slave. To be clear, the enclosing @code{Build} -object must still have a slave object, just as for any other step -- only, in -this step, the slave does not do anything. - -In this example, the step renames a tarball based on the day of the week. - -@example -from buildbot.steps.transfer import FileUpload -from buildbot.steps.master import MasterShellCommand - -f.addStep(FileUpload(slavesrc="widgetsoft.tar.gz", - masterdest="/var/buildoutputs/widgetsoft-new.tar.gz")) -f.addStep(MasterShellCommand(command=""" - cd /var/buildoutputs; - mv widgetsoft-new.tar.gz widgetsoft-`date +%a`.tar.gz""")) -@end example - -@node Triggering Schedulers -@subsection Triggering Schedulers - -The counterpart to the Triggerable described in section -@pxref{Triggerable Scheduler} is the Trigger BuildStep. - -@example -from buildbot.steps.trigger import Trigger -f.addStep(Trigger(schedulerNames=['build-prep'], - waitForFinish=True, - updateSourceStamp=True, - set_properties=@{ 'quick' : False @}, - copy_properties=[ 'release_code_name' ])) -@end example - -The @code{schedulerNames=} argument lists the Triggerables -that should be triggered when this step is executed. Note that -it is possible, but not advisable, to create a cycle where a build -continually triggers itself, because the schedulers are specified -by name. - -If @code{waitForFinish} is True, then the step will not finish until -all of the builds from the triggered schedulers have finished. If this -argument is False (the default) or not given, then the buildstep -succeeds immediately after triggering the schedulers. - -If @code{updateSourceStamp} is True (the default), then step updates -the SourceStamp given to the Triggerables to include -@code{got_revision} (the revision actually used in this build) as -@code{revision} (the revision to use in the triggered builds). This is -useful to ensure that all of the builds use exactly the same -SourceStamp, even if other Changes have occurred while the build was -running. - -Two parameters allow control of the properties that are passed to the triggered -scheduler. To simply copy properties verbatim, list them in the -@code{copy_properties} parameter. To set properties explicitly, use the more -sophisticated @code{set_properties}, which takes a dictionary mapping property -names to values. You may use @code{WithProperties} here to dynamically -construct new property values. - -@node Writing New BuildSteps -@subsection Writing New BuildSteps - -While it is a good idea to keep your build process self-contained in -the source code tree, sometimes it is convenient to put more -intelligence into your Buildbot configuration. One way to do this is -to write a custom BuildStep. Once written, this Step can be used in -the @file{master.cfg} file. - -The best reason for writing a custom BuildStep is to better parse the -results of the command being run. For example, a BuildStep that knows -about JUnit could look at the logfiles to determine which tests had -been run, how many passed and how many failed, and then report more -detailed information than a simple @code{rc==0} -based ``good/bad'' -decision. - -@menu -* Writing BuildStep Constructors:: -* BuildStep LogFiles:: -* Reading Logfiles:: -* Adding LogObservers:: -* BuildStep URLs:: -@end menu - -@node Writing BuildStep Constructors -@subsubsection Writing BuildStep Constructors - -BuildStep classes have some extra equipment, because they are their own -factories. Consider the use of a BuildStep in @file{master.cfg}: - -@example -f.addStep(MyStep(someopt="stuff", anotheropt=1)) -@end example - -This creates a single instance of class @code{MyStep}. However, Buildbot needs -a new object each time the step is executed. this is accomplished by storing -the information required to instantiate a new object in the @code{factory} -attribute. When the time comes to construct a new Build, BuildFactory consults -this attribute (via @code{getStepFactory}) and instantiates a new step object. - -When writing a new step class, then, keep in mind are that you cannot do -anything "interesting" in the constructor -- limit yourself to checking and -storing arguments. To ensure that these arguments are provided to any new -objects, call @code{self.addFactoryArguments} with any keyword arguments your -constructor needs. - -Keep a @code{**kwargs} argument on the end of your options, and pass that up to -the parent class's constructor. - -The whole thing looks like this: - -@example -class Frobinfy(LoggingBuildStep): - def __init__(self, - frob_what="frobee", - frob_how_many=None, - frob_how=None, - **kwargs) - - # check - if frob_how_many is None: - raise TypeError("Frobinfy argument how_many is required") - - # call parent - LoggingBuildStep.__init__(self, **kwargs) - - # and record arguments for later - self.addFactoryArguments( - frob_what=frob_what, - frob_how_many=frob_how_many, - frob_how=frob_how) - -class FastFrobnify(Frobnify): - def __init__(self, - speed=5, - **kwargs) - Frobnify.__init__(self, **kwargs) - self.addFactoryArguments( - speed=speed) -@end example - -@node BuildStep LogFiles -@subsubsection BuildStep LogFiles - -Each BuildStep has a collection of ``logfiles''. Each one has a short -name, like ``stdio'' or ``warnings''. Each LogFile contains an -arbitrary amount of text, usually the contents of some output file -generated during a build or test step, or a record of everything that -was printed to stdout/stderr during the execution of some command. - -These LogFiles are stored to disk, so they can be retrieved later. - -Each can contain multiple ``channels'', generally limited to three -basic ones: stdout, stderr, and ``headers''. For example, when a -ShellCommand runs, it writes a few lines to the ``headers'' channel to -indicate the exact argv strings being run, which directory the command -is being executed in, and the contents of the current environment -variables. Then, as the command runs, it adds a lot of ``stdout'' and -``stderr'' messages. When the command finishes, a final ``header'' -line is added with the exit code of the process. - -Status display plugins can format these different channels in -different ways. For example, the web page shows LogFiles as text/html, -with header lines in blue text, stdout in black, and stderr in red. A -different URL is available which provides a text/plain format, in -which stdout and stderr are collapsed together, and header lines are -stripped completely. This latter option makes it easy to save the -results to a file and run @command{grep} or whatever against the -output. - -Each BuildStep contains a mapping (implemented in a python dictionary) -from LogFile name to the actual LogFile objects. Status plugins can -get a list of LogFiles to display, for example, a list of HREF links -that, when clicked, provide the full contents of the LogFile. - -@heading Using LogFiles in custom BuildSteps - -The most common way for a custom BuildStep to use a LogFile is to -summarize the results of a ShellCommand (after the command has -finished running). For example, a compile step with thousands of lines -of output might want to create a summary of just the warning messages. -If you were doing this from a shell, you would use something like: - -@example -grep "warning:" output.log >warnings.log -@end example - -In a custom BuildStep, you could instead create a ``warnings'' LogFile -that contained the same text. To do this, you would add code to your -@code{createSummary} method that pulls lines from the main output log -and creates a new LogFile with the results: - -@example - def createSummary(self, log): - warnings = [] - for line in log.readlines(): - if "warning:" in line: - warnings.append() - self.addCompleteLog('warnings', "".join(warnings)) -@end example - -This example uses the @code{addCompleteLog} method, which creates a -new LogFile, puts some text in it, and then ``closes'' it, meaning -that no further contents will be added. This LogFile will appear in -the HTML display under an HREF with the name ``warnings'', since that -is the name of the LogFile. - -You can also use @code{addHTMLLog} to create a complete (closed) -LogFile that contains HTML instead of plain text. The normal LogFile -will be HTML-escaped if presented through a web page, but the HTML -LogFile will not. At the moment this is only used to present a pretty -HTML representation of an otherwise ugly exception traceback when -something goes badly wrong during the BuildStep. - -In contrast, you might want to create a new LogFile at the beginning -of the step, and add text to it as the command runs. You can create -the LogFile and attach it to the build by calling @code{addLog}, which -returns the LogFile object. You then add text to this LogFile by -calling methods like @code{addStdout} and @code{addHeader}. When you -are done, you must call the @code{finish} method so the LogFile can be -closed. It may be useful to create and populate a LogFile like this -from a LogObserver method @xref{Adding LogObservers}. - -The @code{logfiles=} argument to @code{ShellCommand} (see -@pxref{ShellCommand}) creates new LogFiles and fills them in realtime -by asking the buildslave to watch a actual file on disk. The -buildslave will look for additions in the target file and report them -back to the BuildStep. These additions will be added to the LogFile by -calling @code{addStdout}. These secondary LogFiles can be used as the -source of a LogObserver just like the normal ``stdio'' LogFile. - -@node Reading Logfiles -@subsubsection Reading Logfiles - -Once a LogFile has been added to a BuildStep with @code{addLog()}, -@code{addCompleteLog()}, @code{addHTMLLog()}, or @code{logfiles=}, -your BuildStep can retrieve it by using @code{getLog()}: - -@example -class MyBuildStep(ShellCommand): - logfiles = @{ "nodelog": "_test/node.log" @} - - def evaluateCommand(self, cmd): - nodelog = self.getLog("nodelog") - if "STARTED" in nodelog.getText(): - return SUCCESS - else: - return FAILURE -@end example - -For a complete list of the methods you can call on a LogFile, please -see the docstrings on the @code{IStatusLog} class in -@file{buildbot/interfaces.py}. - - -@node Adding LogObservers, BuildStep URLs, Reading Logfiles, Writing New BuildSteps -@subsubsection Adding LogObservers - -@cindex LogObserver -@cindex LogLineObserver - -Most shell commands emit messages to stdout or stderr as they operate, -especially if you ask them nicely with a @code{--verbose} flag of some -sort. They may also write text to a log file while they run. Your -BuildStep can watch this output as it arrives, to keep track of how -much progress the command has made. You can get a better measure of -progress by counting the number of source files compiled or test cases -run than by merely tracking the number of bytes that have been written -to stdout. This improves the accuracy and the smoothness of the ETA -display. - -To accomplish this, you will need to attach a @code{LogObserver} to -one of the log channels, most commonly to the ``stdio'' channel but -perhaps to another one which tracks a log file. This observer is given -all text as it is emitted from the command, and has the opportunity to -parse that output incrementally. Once the observer has decided that -some event has occurred (like a source file being compiled), it can -use the @code{setProgress} method to tell the BuildStep about the -progress that this event represents. - -There are a number of pre-built @code{LogObserver} classes that you -can choose from (defined in @code{buildbot.process.buildstep}, and of -course you can subclass them to add further customization. The -@code{LogLineObserver} class handles the grunt work of buffering and -scanning for end-of-line delimiters, allowing your parser to operate -on complete stdout/stderr lines. (Lines longer than a set maximum -length are dropped; the maximum defaults to 16384 bytes, but you can -change it by calling @code{setMaxLineLength()} on your -@code{LogLineObserver} instance. Use @code{sys.maxint} for effective -infinity.) - -For example, let's take a look at the @code{TrialTestCaseCounter}, -which is used by the Trial step to count test cases as they are run. -As Trial executes, it emits lines like the following: - -@example -buildbot.test.test_config.ConfigTest.testDebugPassword ... [OK] -buildbot.test.test_config.ConfigTest.testEmpty ... [OK] -buildbot.test.test_config.ConfigTest.testIRC ... [FAIL] -buildbot.test.test_config.ConfigTest.testLocks ... [OK] -@end example - -When the tests are finished, trial emits a long line of ``======'' and -then some lines which summarize the tests that failed. We want to -avoid parsing these trailing lines, because their format is less -well-defined than the ``[OK]'' lines. - -The parser class looks like this: - -@example -from buildbot.process.buildstep import LogLineObserver - -class TrialTestCaseCounter(LogLineObserver): - _line_re = re.compile(r'^([\w\.]+) \.\.\. \[([^\]]+)\]$') - numTests = 0 - finished = False - - def outLineReceived(self, line): - if self.finished: - return - if line.startswith("=" * 40): - self.finished = True - return - - m = self._line_re.search(line.strip()) - if m: - testname, result = m.groups() - self.numTests += 1 - self.step.setProgress('tests', self.numTests) -@end example - -This parser only pays attention to stdout, since that's where trial -writes the progress lines. It has a mode flag named @code{finished} to -ignore everything after the ``===='' marker, and a scary-looking -regular expression to match each line while hopefully ignoring other -messages that might get displayed as the test runs. - -Each time it identifies a test has been completed, it increments its -counter and delivers the new progress value to the step with -@code{self.step.setProgress}. This class is specifically measuring -progress along the ``tests'' metric, in units of test cases (as -opposed to other kinds of progress like the ``output'' metric, which -measures in units of bytes). The Progress-tracking code uses each -progress metric separately to come up with an overall completion -percentage and an ETA value. - -To connect this parser into the @code{Trial} BuildStep, -@code{Trial.__init__} ends with the following clause: - -@example - # this counter will feed Progress along the 'test cases' metric - counter = TrialTestCaseCounter() - self.addLogObserver('stdio', counter) - self.progressMetrics += ('tests',) -@end example - -This creates a TrialTestCaseCounter and tells the step that the -counter wants to watch the ``stdio'' log. The observer is -automatically given a reference to the step in its @code{.step} -attribute. - -@subheading A Somewhat Whimsical Example - -Let's say that we've got some snazzy new unit-test framework called -Framboozle. It's the hottest thing since sliced bread. It slices, it -dices, it runs unit tests like there's no tomorrow. Plus if your unit -tests fail, you can use its name for a Web 2.1 startup company, make -millions of dollars, and hire engineers to fix the bugs for you, while -you spend your afternoons lazily hang-gliding along a scenic pacific -beach, blissfully unconcerned about the state of your -tests.@footnote{framboozle.com is still available. Remember, I get 10% -:).} - -To run a Framboozle-enabled test suite, you just run the 'framboozler' -command from the top of your source code tree. The 'framboozler' -command emits a bunch of stuff to stdout, but the most interesting bit -is that it emits the line "FNURRRGH!" every time it finishes running a -test case@footnote{Framboozle gets very excited about running unit -tests.}. You'd like to have a test-case counting LogObserver that -watches for these lines and counts them, because counting them will -help the buildbot more accurately calculate how long the build will -take, and this will let you know exactly how long you can sneak out of -the office for your hang-gliding lessons without anyone noticing that -you're gone. - -This will involve writing a new BuildStep (probably named -"Framboozle") which inherits from ShellCommand. The BuildStep class -definition itself will look something like this: - -@example -# START -from buildbot.steps.shell import ShellCommand -from buildbot.process.buildstep import LogLineObserver - -class FNURRRGHCounter(LogLineObserver): - numTests = 0 - def outLineReceived(self, line): - if "FNURRRGH!" in line: - self.numTests += 1 - self.step.setProgress('tests', self.numTests) - -class Framboozle(ShellCommand): - command = ["framboozler"] - - def __init__(self, **kwargs): - ShellCommand.__init__(self, **kwargs) # always upcall! - counter = FNURRRGHCounter()) - self.addLogObserver('stdio', counter) - self.progressMetrics += ('tests',) -# FINISH -@end example - -So that's the code that we want to wind up using. How do we actually -deploy it? - -You have a couple of different options. - -Option 1: The simplest technique is to simply put this text -(everything from START to FINISH) in your master.cfg file, somewhere -before the BuildFactory definition where you actually use it in a -clause like: - -@example -f = BuildFactory() -f.addStep(SVN(svnurl="stuff")) -f.addStep(Framboozle()) -@end example - -Remember that master.cfg is secretly just a python program with one -job: populating the BuildmasterConfig dictionary. And python programs -are allowed to define as many classes as they like. So you can define -classes and use them in the same file, just as long as the class is -defined before some other code tries to use it. - -This is easy, and it keeps the point of definition very close to the -point of use, and whoever replaces you after that unfortunate -hang-gliding accident will appreciate being able to easily figure out -what the heck this stupid "Framboozle" step is doing anyways. The -downside is that every time you reload the config file, the Framboozle -class will get redefined, which means that the buildmaster will think -that you've reconfigured all the Builders that use it, even though -nothing changed. Bleh. - -Option 2: Instead, we can put this code in a separate file, and import -it into the master.cfg file just like we would the normal buildsteps -like ShellCommand and SVN. - -Create a directory named ~/lib/python, put everything from START to -FINISH in ~/lib/python/framboozle.py, and run your buildmaster using: - -@example - PYTHONPATH=~/lib/python buildbot start MASTERDIR -@end example - -or use the @file{Makefile.buildbot} to control the way -@command{buildbot start} works. Or add something like this to -something like your ~/.bashrc or ~/.bash_profile or ~/.cshrc: - -@example - export PYTHONPATH=~/lib/python -@end example - -Once we've done this, our master.cfg can look like: - -@example -from framboozle import Framboozle -f = BuildFactory() -f.addStep(SVN(svnurl="stuff")) -f.addStep(Framboozle()) -@end example - -or: - -@example -import framboozle -f = BuildFactory() -f.addStep(SVN(svnurl="stuff")) -f.addStep(framboozle.Framboozle()) -@end example - -(check out the python docs for details about how "import" and "from A -import B" work). - -What we've done here is to tell python that every time it handles an -"import" statement for some named module, it should look in our -~/lib/python/ for that module before it looks anywhere else. After our -directories, it will try in a bunch of standard directories too -(including the one where buildbot is installed). By setting the -PYTHONPATH environment variable, you can add directories to the front -of this search list. - -Python knows that once it "import"s a file, it doesn't need to -re-import it again. This means that reconfiguring the buildmaster -(with "buildbot reconfig", for example) won't make it think the -Framboozle class has changed every time, so the Builders that use it -will not be spuriously restarted. On the other hand, you either have -to start your buildmaster in a slightly weird way, or you have to -modify your environment to set the PYTHONPATH variable. - - -Option 3: Install this code into a standard python library directory - -Find out what your python's standard include path is by asking it: - -@example -80:warner@@luther% python -Python 2.4.4c0 (#2, Oct 2 2006, 00:57:46) -[GCC 4.1.2 20060928 (prerelease) (Debian 4.1.1-15)] on linux2 -Type "help", "copyright", "credits" or "license" for more information. ->>> import sys ->>> import pprint ->>> pprint.pprint(sys.path) -['', - '/usr/lib/python24.zip', - '/usr/lib/python2.4', - '/usr/lib/python2.4/plat-linux2', - '/usr/lib/python2.4/lib-tk', - '/usr/lib/python2.4/lib-dynload', - '/usr/local/lib/python2.4/site-packages', - '/usr/lib/python2.4/site-packages', - '/usr/lib/python2.4/site-packages/Numeric', - '/var/lib/python-support/python2.4', - '/usr/lib/site-python'] -@end example - -In this case, putting the code into -/usr/local/lib/python2.4/site-packages/framboozle.py would work just -fine. We can use the same master.cfg "import framboozle" statement as -in Option 2. By putting it in a standard include directory (instead of -the decidedly non-standard ~/lib/python), we don't even have to set -PYTHONPATH to anything special. The downside is that you probably have -to be root to write to one of those standard include directories. - - -Option 4: Submit the code for inclusion in the Buildbot distribution - -Make a fork of buildbot on http://github.com/djmitche/buildbot or post a patch -in a bug at http://buildbot.net. In either case, post a note about your patch -to the mailing list, so others can provide feedback and, eventually, commit it. - -@example -from buildbot.steps import framboozle -f = BuildFactory() -f.addStep(SVN(svnurl="stuff")) -f.addStep(framboozle.Framboozle()) -@end example - -And then you don't even have to install framboozle.py anywhere on your -system, since it will ship with Buildbot. You don't have to be root, -you don't have to set PYTHONPATH. But you do have to make a good case -for Framboozle being worth going into the main distribution, you'll -probably have to provide docs and some unit test cases, you'll need to -figure out what kind of beer the author likes, and then you'll have to -wait until the next release. But in some environments, all this is -easier than getting root on your buildmaster box, so the tradeoffs may -actually be worth it. - - - -Putting the code in master.cfg (1) makes it available to that -buildmaster instance. Putting it in a file in a personal library -directory (2) makes it available for any buildmasters you might be -running. Putting it in a file in a system-wide shared library -directory (3) makes it available for any buildmasters that anyone on -that system might be running. Getting it into the buildbot's upstream -repository (4) makes it available for any buildmasters that anyone in -the world might be running. It's all a matter of how widely you want -to deploy that new class. - - - -@node BuildStep URLs, , Adding LogObservers, Writing New BuildSteps -@subsubsection BuildStep URLs - -@cindex links -@cindex BuildStep URLs -@cindex addURL - -Each BuildStep has a collection of ``links''. Like its collection of -LogFiles, each link has a name and a target URL. The web status page -creates HREFs for each link in the same box as it does for LogFiles, -except that the target of the link is the external URL instead of an -internal link to a page that shows the contents of the LogFile. - -These external links can be used to point at build information hosted -on other servers. For example, the test process might produce an -intricate description of which tests passed and failed, or some sort -of code coverage data in HTML form, or a PNG or GIF image with a graph -of memory usage over time. The external link can provide an easy way -for users to navigate from the buildbot's status page to these -external web sites or file servers. Note that the step itself is -responsible for insuring that there will be a document available at -the given URL (perhaps by using @command{scp} to copy the HTML output -to a @file{~/public_html/} directory on a remote web server). Calling -@code{addURL} does not magically populate a web server. - -To set one of these links, the BuildStep should call the @code{addURL} -method with the name of the link and the target URL. Multiple URLs can -be set. - -In this example, we assume that the @command{make test} command causes -a collection of HTML files to be created and put somewhere on the -coverage.example.org web server, in a filename that incorporates the -build number. - -@example -class TestWithCodeCoverage(BuildStep): - command = ["make", "test", - WithProperties("buildnum=%s", "buildnumber")] - - def createSummary(self, log): - buildnumber = self.getProperty("buildnumber") - url = "http://coverage.example.org/builds/%s.html" % buildnumber - self.addURL("coverage", url) -@end example - -You might also want to extract the URL from some special message -output by the build process itself: - -@example -class TestWithCodeCoverage(BuildStep): - command = ["make", "test", - WithProperties("buildnum=%s", "buildnumber")] - - def createSummary(self, log): - output = StringIO(log.getText()) - for line in output.readlines(): - if line.startswith("coverage-url:"): - url = line[len("coverage-url:"):].strip() - self.addURL("coverage", url) - return -@end example - -Note that a build process which emits both stdout and stderr might -cause this line to be split or interleaved between other lines. It -might be necessary to restrict the getText() call to only stdout with -something like this: - -@example - output = StringIO("".join([c[1] - for c in log.getChunks() - if c[0] == LOG_CHANNEL_STDOUT])) -@end example - -Of course if the build is run under a PTY, then stdout and stderr will -be merged before the buildbot ever sees them, so such interleaving -will be unavoidable. - - -@node Interlocks, Build Factories, Build Steps, Build Process -@section Interlocks - -@cindex locks -@slindex buildbot.locks.MasterLock -@slindex buildbot.locks.SlaveLock -@slindex buildbot.locks.LockAccess - -Until now, we assumed that a master can run builds at any slave whenever -needed or desired. Some times, you want to enforce additional constraints on -builds. For reasons like limited network bandwidth, old slave machines, or a -self-willed data base server, you may want to limit the number of builds (or -build steps) that can access a resource. - -The mechanism used by Buildbot is known as the read/write lock.@footnote{See -http://en.wikipedia.org/wiki/Read/write_lock_pattern for more information.} It -allows either many readers or a single writer but not a combination of readers -and writers. The general lock has been modified and extended for use in -Buildbot. Firstly, the general lock allows an infinite number of readers. In -Buildbot, we often want to put an upper limit on the number of readers, for -example allowing two out of five possible builds at the same time. To do this, -the lock counts the number of active readers. Secondly, the terms @emph{read -mode} and @emph{write mode} are confusing in Buildbot context. They have been -replaced by @emph{counting mode} (since the lock counts them) and @emph{exclusive -mode}. As a result of these changes, locks in Buildbot allow a number of -builds (upto some fixed number) in counting mode, or they allow one build in -exclusive mode. - -Often, not all slaves are equal. To allow for this situation, Buildbot allows -to have a separate upper limit on the count for each slave. In this way, you -can have at most 3 concurrent builds at a fast slave, 2 at a slightly older -slave, and 1 at all other slaves. - -The final thing you can specify when you introduce a new lock is its scope. -Some constraints are global -- they must be enforced over all slaves. Other -constraints are local to each slave. A @emph{master lock} is used for the -global constraints. You can ensure for example that at most one build (of all -builds running at all slaves) accesses the data base server. With a -@emph{slave lock} you can add a limit local to each slave. With such a lock, -you can for example enforce an upper limit to the number of active builds at a -slave, like above. - -Time for a few examples. Below a master lock is defined to protect a data base, -and a slave lock is created to limit the number of builds at each slave. - -@example -from buildbot import locks - -db_lock = locks.MasterLock("database") -build_lock = locks.SlaveLock("slave_builds", - maxCount = 1, - maxCountForSlave = @{ 'fast': 3, 'new': 2 @}) -@end example - -After importing locks from buildbot, @code{db_lock} is defined to be a master -lock. The @code{"database"} string is used for uniquely identifying the lock. -At the next line, a slave lock called @code{build_lock} is created. It is -identified by the @code{"slave_builds"} string. Since the requirements of the -lock are a bit more complicated, two optional arguments are also specified. The -@code{maxCount} parameter sets the default limit for builds in counting mode to -@code{1}. For the slave called @code{'fast'} however, we want to have at most -three builds, and for the slave called @code{'new'} the upper limit is two -builds running at the same time. - -The next step is using the locks in builds. Buildbot allows a lock to be used -during an entire build (from beginning to end), or only during a single build -step. In the latter case, the lock is claimed for use just before the step -starts, and released again when the step ends. To prevent -deadlocks,@footnote{Deadlock is the situation where two or more slaves each -hold a lock in exclusive mode, and in addition want to claim the lock held by -the other slave exclusively as well. Since locks allow at most one exclusive -user, both slaves will wait forever.} it is not possible to claim or release -locks at other times. - -To use locks, you should add them with a @code{locks} argument. -Each use of a lock is either in counting mode (that is, possibly shared with -other builds) or in exclusive mode. A build or build step proceeds only when it -has acquired all locks. If a build or step needs a lot of locks, it may be -starved@footnote{Starving is the situation that only a few locks are available, -and they are immediately grabbed by another build. As a result, it may take a -long time before all locks needed by the starved build are free at the same -time.} by other builds that need fewer locks. - -To illustrate use of locks, a few examples. - -@example -from buildbot import locks -from buildbot.steps import source, shell -from buildbot.process import factory - -db_lock = locks.MasterLock("database") -build_lock = locks.SlaveLock("slave_builds", - maxCount = 1, - maxCountForSlave = @{ 'fast': 3, 'new': 2 @}) - -f = factory.BuildFactory() -f.addStep(source.SVN(svnurl="http://example.org/svn/Trunk")) -f.addStep(shell.ShellCommand(command="make all")) -f.addStep(shell.ShellCommand(command="make test", - locks=[db_lock.access('exclusive')])) - -b1 = @{'name': 'full1', 'slavename': 'fast', 'builddir': 'f1', 'factory': f, - 'locks': [build_lock.access('counting')] @} - -b2 = @{'name': 'full2', 'slavename': 'new', 'builddir': 'f2', 'factory': f. - 'locks': [build_lock.access('counting')] @} - -b3 = @{'name': 'full3', 'slavename': 'old', 'builddir': 'f3', 'factory': f. - 'locks': [build_lock.access('counting')] @} - -b4 = @{'name': 'full4', 'slavename': 'other', 'builddir': 'f4', 'factory': f. - 'locks': [build_lock.access('counting')] @} - -c['builders'] = [b1, b2, b3, b4] -@end example - -Here we have four slaves @code{b1}, @code{b2}, @code{b3}, and @code{b4}. Each -slave performs the same checkout, make, and test build step sequence. -We want to enforce that at most one test step is executed between all slaves due -to restrictions with the data base server. This is done by adding the -@code{locks=} parameter with the third step. It takes a list of locks with their -access mode. In this case only the @code{db_lock} is needed. The exclusive -access mode is used to ensure there is at most one slave that executes the test -step. - -In addition to exclusive accessing the data base, we also want slaves to stay -responsive even under the load of a large number of builds being triggered. -For this purpose, the slave lock called @code{build_lock} is defined. Since -the restraint holds for entire builds, the lock is specified in the builder -with @code{'locks': [build_lock.access('counting')]}. -@node Build Factories, , Interlocks, Build Process -@section Build Factories - - -Each Builder is equipped with a ``build factory'', which is -responsible for producing the actual @code{Build} objects that perform -each build. This factory is created in the configuration file, and -attached to a Builder through the @code{factory} element of its -dictionary. - -The standard @code{BuildFactory} object creates @code{Build} objects -by default. These Builds will each execute a collection of BuildSteps -in a fixed sequence. Each step can affect the results of the build, -but in general there is little intelligence to tie the different steps -together. You can create subclasses of @code{Build} to implement more -sophisticated build processes, and then use a subclass of -@code{BuildFactory} (or simply set the @code{buildClass} attribute) to -create instances of your new Build subclass. - - -@menu -* BuildStep Objects:: -* BuildFactory:: -* Process-Specific build factories:: -@end menu - -@node BuildStep Objects, BuildFactory, Build Factories, Build Factories -@subsection BuildStep Objects - -The steps used by these builds are all subclasses of @code{BuildStep}. -The standard ones provided with Buildbot are documented later, -@xref{Build Steps}. You can also write your own subclasses to use in -builds. - -The basic behavior for a @code{BuildStep} is to: - -@itemize @bullet -@item -run for a while, then stop -@item -possibly invoke some RemoteCommands on the attached build slave -@item -possibly produce a set of log files -@item -finish with a status described by one of four values defined in -buildbot.status.builder: SUCCESS, WARNINGS, FAILURE, SKIPPED -@item -provide a list of short strings to describe the step -@item -define a color (generally green, orange, or red) with which the -step should be displayed -@end itemize - - -More sophisticated steps may produce additional information and -provide it to later build steps, or store it in the factory to provide -to later builds. - - -@menu -* BuildFactory Attributes:: -* Quick builds:: -@end menu - -@node BuildFactory, Process-Specific build factories, BuildStep Objects, Build Factories -@subsection BuildFactory - -@bfindex buildbot.process.factory.BuildFactory -@bfindex buildbot.process.factory.BasicBuildFactory -@c TODO: what is BasicSVN anyway? -@bfindex buildbot.process.factory.BasicSVN - -The default @code{BuildFactory}, provided in the -@code{buildbot.process.factory} module, contains an internal list of -``BuildStep specifications'': a list of @code{(step_class, kwargs)} -tuples for each. These specification tuples are constructed when the -config file is read, by asking the instances passed to @code{addStep} -for their subclass and arguments. - -When asked to create a Build, the @code{BuildFactory} puts a copy of -the list of step specifications into the new Build object. When the -Build is actually started, these step specifications are used to -create the actual set of BuildSteps, which are then executed one at a -time. This serves to give each Build an independent copy of each step. -For example, a build which consists of a CVS checkout followed by a -@code{make build} would be constructed as follows: - -@example -from buildbot.steps import source, shell -from buildbot.process import factory - -f = factory.BuildFactory() -f.addStep(source.CVS(cvsroot=CVSROOT, cvsmodule="project", mode="update")) -f.addStep(shell.Compile(command=["make", "build"])) -@end example - -(To support config files from buildbot-0.7.5 and earlier, -@code{addStep} also accepts the @code{f.addStep(shell.Compile, -command=["make","build"])} form, although its use is discouraged -because then the @code{Compile} step doesn't get to validate or -complain about its arguments until build time. The modern -pass-by-instance approach allows this validation to occur while the -config file is being loaded, where the admin has a better chance of -noticing problems). - -It is also possible to pass a list of steps into the -@code{BuildFactory} when it is created. Using @code{addStep} is -usually simpler, but there are cases where is is more convenient to -create the list of steps ahead of time.: - -@example -from buildbot.steps import source, shell -from buildbot.process import factory - -all_steps = [source.CVS(cvsroot=CVSROOT, cvsmodule="project", mode="update"), - shell.Compile(command=["make", "build"]), - ] -f = factory.BuildFactory(all_steps) -@end example - - -Each step can affect the build process in the following ways: - -@itemize @bullet -@item -If the step's @code{haltOnFailure} attribute is True, then a failure -in the step (i.e. if it completes with a result of FAILURE) will cause -the whole build to be terminated immediately: no further steps will be -executed, with the exception of steps with @code{alwaysRun} set to -True. @code{haltOnFailure} is useful for setup steps upon which the -rest of the build depends: if the CVS checkout or @code{./configure} -process fails, there is no point in trying to compile or test the -resulting tree. - -@item -If the step's @code{alwaysRun} attribute is True, then it will always -be run, regardless of if previous steps have failed. This is useful -for cleanup steps that should always be run to return the build -directory or build slave into a good state. - -@item -If the @code{flunkOnFailure} or @code{flunkOnWarnings} flag is set, -then a result of FAILURE or WARNINGS will mark the build as a whole as -FAILED. However, the remaining steps will still be executed. This is -appropriate for things like multiple testing steps: a failure in any -one of them will indicate that the build has failed, however it is -still useful to run them all to completion. - -@item -Similarly, if the @code{warnOnFailure} or @code{warnOnWarnings} flag -is set, then a result of FAILURE or WARNINGS will mark the build as -having WARNINGS, and the remaining steps will still be executed. This -may be appropriate for certain kinds of optional build or test steps. -For example, a failure experienced while building documentation files -should be made visible with a WARNINGS result but not be serious -enough to warrant marking the whole build with a FAILURE. - -@end itemize - -In addition, each Step produces its own results, may create logfiles, -etc. However only the flags described above have any effect on the -build as a whole. - -The pre-defined BuildSteps like @code{CVS} and @code{Compile} have -reasonably appropriate flags set on them already. For example, without -a source tree there is no point in continuing the build, so the -@code{CVS} class has the @code{haltOnFailure} flag set to True. Look -in @file{buildbot/steps/*.py} to see how the other Steps are -marked. - -Each Step is created with an additional @code{workdir} argument that -indicates where its actions should take place. This is specified as a -subdirectory of the slave builder's base directory, with a default -value of @code{build}. This is only implemented as a step argument (as -opposed to simply being a part of the base directory) because the -CVS/SVN steps need to perform their checkouts from the parent -directory. - -@menu -* BuildFactory Attributes:: -* Quick builds:: -@end menu - -@node BuildFactory Attributes, Quick builds, BuildFactory, BuildFactory -@subsubsection BuildFactory Attributes - -Some attributes from the BuildFactory are copied into each Build. - -@cindex treeStableTimer - -@table @code -@item useProgress -(defaults to True): if True, the buildmaster keeps track of how long -each step takes, so it can provide estimates of how long future builds -will take. If builds are not expected to take a consistent amount of -time (such as incremental builds in which a random set of files are -recompiled or tested each time), this should be set to False to -inhibit progress-tracking. - -@end table - - -@node Quick builds, , BuildFactory Attributes, BuildFactory -@subsubsection Quick builds - -@bfindex buildbot.process.factory.QuickBuildFactory - -The difference between a ``full build'' and a ``quick build'' is that -quick builds are generally done incrementally, starting with the tree -where the previous build was performed. That simply means that the -source-checkout step should be given a @code{mode='update'} flag, to -do the source update in-place. - -In addition to that, the @code{useProgress} flag should be set to -False. Incremental builds will (or at least the ought to) compile as -few files as necessary, so they will take an unpredictable amount of -time to run. Therefore it would be misleading to claim to predict how -long the build will take. - - -@node Process-Specific build factories, , BuildFactory, Build Factories -@subsection Process-Specific build factories - -Many projects use one of a few popular build frameworks to simplify -the creation and maintenance of Makefiles or other compilation -structures. Buildbot provides several pre-configured BuildFactory -subclasses which let you build these projects with a minimum of fuss. - -@menu -* GNUAutoconf:: -* CPAN:: -* Python distutils:: -* Python/Twisted/trial projects:: -@end menu - -@node GNUAutoconf, CPAN, Process-Specific build factories, Process-Specific build factories -@subsubsection GNUAutoconf - -@bfindex buildbot.process.factory.GNUAutoconf - -@uref{http://www.gnu.org/software/autoconf/, GNU Autoconf} is a -software portability tool, intended to make it possible to write -programs in C (and other languages) which will run on a variety of -UNIX-like systems. Most GNU software is built using autoconf. It is -frequently used in combination with GNU automake. These tools both -encourage a build process which usually looks like this: - -@example -% CONFIG_ENV=foo ./configure --with-flags -% make all -% make check -# make install -@end example - -(except of course the Buildbot always skips the @code{make install} -part). - -The Buildbot's @code{buildbot.process.factory.GNUAutoconf} factory is -designed to build projects which use GNU autoconf and/or automake. The -configuration environment variables, the configure flags, and command -lines used for the compile and test are all configurable, in general -the default values will be suitable. - -Example: - -@example -# use the s() convenience function defined earlier -f = factory.GNUAutoconf(source=s(source.SVN, svnurl=URL, mode="copy"), - flags=["--disable-nls"]) -@end example - -Required Arguments: - -@table @code -@item source -This argument must be a step specification tuple that provides a -BuildStep to generate the source tree. -@end table - -Optional Arguments: - -@table @code -@item configure -The command used to configure the tree. Defaults to -@code{./configure}. Accepts either a string or a list of shell argv -elements. - -@item configureEnv -The environment used for the initial configuration step. This accepts -a dictionary which will be merged into the buildslave's normal -environment. This is commonly used to provide things like -@code{CFLAGS="-O2 -g"} (to turn off debug symbols during the compile). -Defaults to an empty dictionary. - -@item configureFlags -A list of flags to be appended to the argument list of the configure -command. This is commonly used to enable or disable specific features -of the autoconf-controlled package, like @code{["--without-x"]} to -disable windowing support. Defaults to an empty list. - -@item compile -this is a shell command or list of argv values which is used to -actually compile the tree. It defaults to @code{make all}. If set to -None, the compile step is skipped. - -@item test -this is a shell command or list of argv values which is used to run -the tree's self-tests. It defaults to @code{make check}. If set to -None, the test step is skipped. - -@end table - - -@node CPAN, Python distutils, GNUAutoconf, Process-Specific build factories -@subsubsection CPAN - -@bfindex buildbot.process.factory.CPAN - -Most Perl modules available from the @uref{http://www.cpan.org/, CPAN} -archive use the @code{MakeMaker} module to provide configuration, -build, and test services. The standard build routine for these modules -looks like: - -@example -% perl Makefile.PL -% make -% make test -# make install -@end example - -(except again Buildbot skips the install step) - -Buildbot provides a @code{CPAN} factory to compile and test these -projects. - - -Arguments: -@table @code -@item source -(required): A step specification tuple, like that used by GNUAutoconf. - -@item perl -A string which specifies the @code{perl} executable to use. Defaults -to just @code{perl}. - -@end table - - -@node Python distutils, Python/Twisted/trial projects, CPAN, Process-Specific build factories -@subsubsection Python distutils - -@bfindex buildbot.process.factory.Distutils - -Most Python modules use the @code{distutils} package to provide -configuration and build services. The standard build process looks -like: - -@example -% python ./setup.py build -% python ./setup.py install -@end example - -Unfortunately, although Python provides a standard unit-test framework -named @code{unittest}, to the best of my knowledge @code{distutils} -does not provide a standardized target to run such unit tests. (Please -let me know if I'm wrong, and I will update this factory.) - -The @code{Distutils} factory provides support for running the build -part of this process. It accepts the same @code{source=} parameter as -the other build factories. - - -Arguments: -@table @code -@item source -(required): A step specification tuple, like that used by GNUAutoconf. - -@item python -A string which specifies the @code{python} executable to use. Defaults -to just @code{python}. - -@item test -Provides a shell command which runs unit tests. This accepts either a -string or a list. The default value is None, which disables the test -step (since there is no common default command to run unit tests in -distutils modules). - -@end table - - -@node Python/Twisted/trial projects, , Python distutils, Process-Specific build factories -@subsubsection Python/Twisted/trial projects - -@bfindex buildbot.process.factory.Trial -@c TODO: document these steps better -@bsindex buildbot.steps.python_twisted.HLint -@bsindex buildbot.steps.python_twisted.Trial -@bsindex buildbot.steps.python_twisted.ProcessDocs -@bsindex buildbot.steps.python_twisted.BuildDebs -@bsindex buildbot.steps.python_twisted.RemovePYCs - -Twisted provides a unit test tool named @code{trial} which provides a -few improvements over Python's built-in @code{unittest} module. Many -python projects which use Twisted for their networking or application -services also use trial for their unit tests. These modules are -usually built and tested with something like the following: - -@example -% python ./setup.py build -% PYTHONPATH=build/lib.linux-i686-2.3 trial -v PROJECTNAME.test -% python ./setup.py install -@end example - -Unfortunately, the @file{build/lib} directory into which the -built/copied .py files are placed is actually architecture-dependent, -and I do not yet know of a simple way to calculate its value. For many -projects it is sufficient to import their libraries ``in place'' from -the tree's base directory (@code{PYTHONPATH=.}). - -In addition, the @var{PROJECTNAME} value where the test files are -located is project-dependent: it is usually just the project's -top-level library directory, as common practice suggests the unit test -files are put in the @code{test} sub-module. This value cannot be -guessed, the @code{Trial} class must be told where to find the test -files. - -The @code{Trial} class provides support for building and testing -projects which use distutils and trial. If the test module name is -specified, trial will be invoked. The library path used for testing -can also be set. - -One advantage of trial is that the Buildbot happens to know how to -parse trial output, letting it identify which tests passed and which -ones failed. The Buildbot can then provide fine-grained reports about -how many tests have failed, when individual tests fail when they had -been passing previously, etc. - -Another feature of trial is that you can give it a series of source -.py files, and it will search them for special @code{test-case-name} -tags that indicate which test cases provide coverage for that file. -Trial can then run just the appropriate tests. This is useful for -quick builds, where you want to only run the test cases that cover the -changed functionality. - -Arguments: -@table @code -@item testpath -Provides a directory to add to @code{PYTHONPATH} when running the unit -tests, if tests are being run. Defaults to @code{.} to include the -project files in-place. The generated build library is frequently -architecture-dependent, but may simply be @file{build/lib} for -pure-python modules. - -@item python -which python executable to use. This list will form the start of -the argv array that will launch trial. If you use this, -you should set @code{trial} to an explicit path (like -@code{/usr/bin/trial} or @code{./bin/trial}). The parameter defaults to @code{None}, which -leaves it out entirely (running @code{trial args} instead of -@code{python ./bin/trial args}). Likely values are @code{['python']}, -@code{['python2.2']}, or @code{['python', '-Wall']}. - -@item trial -provides the name of the @code{trial} command. It is occasionally -useful to use an alternate executable, such as @code{trial2.2} which -might run the tests under an older version of Python. Defaults to -@code{trial}. - -@item trialMode -a list of arguments to pass to trial, specifically to set the reporting mode. -This defaults to @code{['--reporter=bwverbose']}, which only works for -Twisted-2.1.0 and later. - -@item trialArgs -a list of arguments to pass to trial, available to turn on any extra flags you -like. Defaults to @code{[]}. - -@item tests -Provides a module name or names which contain the unit tests for this -project. Accepts a string, typically @code{PROJECTNAME.test}, or a -list of strings. Defaults to None, indicating that no tests should be -run. You must either set this or @code{testChanges}. - -@item testChanges -if True, ignore the @code{tests} parameter and instead ask the Build for all -the files that make up the Changes going into this build. Pass these filenames -to trial and ask it to look for test-case-name tags, running just the tests -necessary to cover the changes. - -@item recurse -If @code{True}, tells Trial (with the @code{--recurse} argument) to look in all -subdirectories for additional test cases. - -@item reactor -which reactor to use, like 'gtk' or 'java'. If not provided, the Twisted's -usual platform-dependent default is used. - -@item randomly -If @code{True}, tells Trial (with the @code{--random=0} argument) to -run the test cases in random order, which sometimes catches subtle -inter-test dependency bugs. Defaults to @code{False}. -@end table - -The step can also take any of the @code{ShellCommand} arguments, e.g., -@code{haltOnFailure}. - -Unless one of @code{tests} or @code{testChanges} are set, the step will -generate an exception. - -@node Status Delivery, Command-line tool, Build Process, Top -@chapter Status Delivery - -More details are available in the docstrings for each class, use a -command like @code{pydoc buildbot.status.html.WebStatus} to see them. -Most status delivery objects take a @code{categories=} argument, which -can contain a list of ``category'' names: in this case, it will only -show status for Builders that are in one of the named categories. - -(implementor's note: each of these objects should be a -service.MultiService which will be attached to the BuildMaster object -when the configuration is processed. They should use -@code{self.parent.getStatus()} to get access to the top-level IStatus -object, either inside @code{startService} or later. They may call -@code{status.subscribe()} in @code{startService} to receive -notifications of builder events, in which case they must define -@code{builderAdded} and related methods. See the docstrings in -@file{buildbot/interfaces.py} for full details.) - -@menu -* WebStatus:: -* MailNotifier:: -* IRC Bot:: -* PBListener:: -* Writing New Status Plugins:: -@end menu - -@c @node Email Delivery, , Status Delivery, Status Delivery -@c @subsection Email Delivery - -@c DOCUMENT THIS - - -@node WebStatus, MailNotifier, Status Delivery, Status Delivery -@section WebStatus - -@cindex WebStatus -@stindex buildbot.status.web.baseweb.WebStatus - -The @code{buildbot.status.html.WebStatus} status target runs a small -web server inside the buildmaster. You can point a browser at this web -server and retrieve information about every build the buildbot knows -about, as well as find out what the buildbot is currently working on. - -The first page you will see is the ``Welcome Page'', which contains -links to all the other useful pages. This page is simply served from -the @file{public_html/index.html} file in the buildmaster's base -directory, where it is created by the @command{buildbot create-master} -command along with the rest of the buildmaster. - -The most complex resource provided by @code{WebStatus} is the -``Waterfall Display'', which shows a time-based chart of events. This -somewhat-busy display provides detailed information about all steps of -all recent builds, and provides hyperlinks to look at individual build -logs and source changes. By simply reloading this page on a regular -basis, you will see a complete description of everything the buildbot -is currently working on. - -A similar, but more developer-oriented display is the "Grid" display. This -arranges builds by SourceStamp (horizontal axis) and builder (vertical axis), -and can provide quick information as to which revisions are passing or failing -on which builders. - -There are also pages with more specialized information. For example, -there is a page which shows the last 20 builds performed by the -buildbot, one line each. Each line is a link to detailed information -about that build. By adding query arguments to the URL used to reach -this page, you can narrow the display to builds that involved certain -branches, or which ran on certain Builders. These pages are described -in great detail below. - - -When the buildmaster is created, a subdirectory named -@file{public_html/} is created in its base directory. By default, @code{WebStatus} -will serve files from this directory: for example, when a user points -their browser at the buildbot's @code{WebStatus} URL, they will see -the contents of the @file{public_html/index.html} file. Likewise, -@file{public_html/robots.txt}, @file{public_html/buildbot.css}, and -@file{public_html/favicon.ico} are all useful things to have in there. -The first time a buildmaster is created, the @file{public_html} -directory is populated with some sample files, which you will probably -want to customize for your own project. These files are all static: -the buildbot does not modify them in any way as it serves them to HTTP -clients. - -@example -from buildbot.status.html import WebStatus -c['status'].append(WebStatus(8080)) -@end example - -Note that the initial robots.txt file has Disallow lines for all of -the dynamically-generated buildbot pages, to discourage web spiders -and search engines from consuming a lot of CPU time as they crawl -through the entire history of your buildbot. If you are running the -buildbot behind a reverse proxy, you'll probably need to put the -robots.txt file somewhere else (at the top level of the parent web -server), and replace the URL prefixes in it with more suitable values. - -If you would like to use an alternative root directory, add the -@code{public_html=..} option to the @code{WebStatus} creation: - -@example -c['status'].append(WebStatus(8080, public_html="/var/www/buildbot")) -@end example - -In addition, if you are familiar with twisted.web @emph{Resource -Trees}, you can write code to add additional pages at places inside -this web space. Just use @code{webstatus.putChild} to place these -resources. - -The following section describes the special URLs and the status views -they provide. - - -@menu -* Buildbot Web Resources:: -* WebStatus Configuration Parameters:: -* XMLRPC server:: -@end menu - -@node Buildbot Web Resources -@subsection Buildbot Web Resources - -Certain URLs are ``magic'', and the pages they serve are created by -code in various classes in the @file{buildbot.status.web} package -instead of being read from disk. The most common way to access these -pages is for the buildmaster admin to write or modify the -@file{index.html} page to contain links to them. Of course other -project web pages can contain links to these buildbot pages as well. - -Many pages can be modified by adding query arguments to the URL. For -example, a page which shows the results of the most recent build -normally does this for all builders at once. But by appending -``?builder=i386'' to the end of the URL, the page will show only the -results for the ``i386'' builder. When used in this way, you can add -multiple ``builder='' arguments to see multiple builders. Remembering -that URL query arguments are separated @emph{from each other} with -ampersands, a URL that ends in ``?builder=i386&builder=ppc'' would -show builds for just those two Builders. - -The @code{branch=} query argument can be used on some pages. This -filters the information displayed by that page down to only the builds -or changes which involved the given branch. Use @code{branch=trunk} to -reference the trunk: if you aren't intentionally using branches, -you're probably using trunk. Multiple @code{branch=} arguments can be -used to examine multiple branches at once (so appending -@code{?branch=foo&branch=bar} to the URL will show builds involving -either branch). No @code{branch=} arguments means to show builds and -changes for all branches. - -Some pages may include the Builder name or the build number in the -main part of the URL itself. For example, a page that describes Build -#7 of the ``i386'' builder would live at @file{/builders/i386/builds/7}. - -The table below lists all of the internal pages and the URLs that can -be used to access them. - -@table @code - -@item /waterfall - -This provides a chronologically-oriented display of the activity of -all builders. It is the same display used by the Waterfall display. - -By adding one or more ``builder='' query arguments, the Waterfall is -restricted to only showing information about the given Builders. By -adding one or more ``branch='' query arguments, the display is -restricted to showing information about the given branches. In -addition, adding one or more ``category='' query arguments to the URL -will limit the display to Builders that were defined with one of the -given categories. - -A 'show_events=true' query argument causes the display to include -non-Build events, like slaves attaching and detaching, as well as -reconfiguration events. 'show_events=false' hides these events. The -default is to show them. - -By adding the 'failures_only=true' query argument, the Waterfall is -restricted to only showing information about the builders that -are currently failing. A builder is considered failing if the -last finished build was not successful, a step in the current -build(s) is failing, or if the builder is offline. - -The @code{last_time=}, @code{first_time=}, and @code{show_time=} -arguments will control what interval of time is displayed. The default -is to show the latest events, but these can be used to look at earlier -periods in history. The @code{num_events=} argument also provides a -limit on the size of the displayed page. - -The Waterfall has references to resources many of the other portions -of the URL space: @file{/builders} for access to individual builds, -@file{/changes} for access to information about source code changes, -etc. - -@item /grid - -This provides a chronologically oriented display of builders, by -revision. The builders are listed down the left side of the page, -and the revisions are listed across the top. - -By adding one ore more ``category='' arguments the grid will be -restricted to revisions in those categories. - -A ``width=N'' argument will limit the number of revisions shown to N, -defaulting to 5. - -A ``branch=BRANCHNAME'' argument will limit the grid to revisions on -branch BRANCHNAME. - -@item /tgrid - -The Transposed Grid is similar to the standard grid, but, as the name -implies, transposes the grid: the revisions are listed down the left side -of the page, and the build hosts are listed across the top. It accepts -the same query arguments. The exception being that instead of ``width'' -the argument is named ``length.'' - -@item /console - -EXPERIMENTAL: This provides a developer-oriented display of the the last -changes and how they affected the builders. - -It allows a developer to quickly see the status of each builder for the -first build including his or her change. A green box means that the change -succeeded for all the steps for a given builder. A red box means that -the changed introduced a new regression on a builder. An orange box -means that at least one of the test failed, but it was also failing -in the previous build, so it is not possible to see if there was any -regressions from this change. Finally a yellow box means that the test -is in progress. - -By adding one or more ``builder='' query arguments, the Console view is -restricted to only showing information about the given Builders. By -adding one or more ``branch='' query arguments, the display is -restricted to showing information about the given branches. In -addition, adding one or more ``category='' query arguments to the URL -will limit the display to Builders that were defined with one of the -given categories. - -By adding one or more ``name='' query arguments to the URL, the console view is -restricted to only showing changes made by the given users. - -NOTE: To use this page, your buildbot.css file in public_html -must be the one found in buildbot/status/web/extended.css. - -The console view is still in development. At this moment it supports -only the source control managers that have an integer based revision id, -like svn. It also has some issues with displaying multiple braches at the -same time. If you do have multiple branches, you should use the -``branch='' query argument. The @code{order_console_by_time} option may help -with non-integer VCs: - -@example -w = html.WebStatus(http_port=8080, order_console_by_time=True) -@end example - -@item /rss - -This provides a rss feed summarizing all failed builds. The same -query-arguments used by 'waterfall' can be added to filter the -feed output. - -@item /atom - -This provides an atom feed summarizing all failed builds. The same -query-arguments used by 'waterfall' can be added to filter the feed -output. - -@item /json - -This view provides quick access to Buildbot status information in a form that -is easiliy digested from other programs, including JavaScript. See -@code{/json/help} for detailed interactive documentation of the output formats -for this view. - -@item /buildstatus?builder=$BUILDERNAME&number=$BUILDNUM - -This displays a waterfall-like chronologically-oriented view of all the -steps for a given build number on a given builder. - -@item /builders/$BUILDERNAME - -This describes the given Builder, and provides buttons to force a -build. A @code{numbuilds=} argument will control how many build lines -are displayed (5 by default). - -@item /builders/$BUILDERNAME/builds/$BUILDNUM - -This describes a specific Build. - -@item /builders/$BUILDERNAME/builds/$BUILDNUM/steps/$STEPNAME - -This describes a specific BuildStep. - -@item /builders/$BUILDERNAME/builds/$BUILDNUM/steps/$STEPNAME/logs/$LOGNAME - -This provides an HTML representation of a specific logfile. - -@item /builders/$BUILDERNAME/builds/$BUILDNUM/steps/$STEPNAME/logs/$LOGNAME/text - -This returns the logfile as plain text, without any HTML coloring -markup. It also removes the ``headers'', which are the lines that -describe what command was run and what the environment variable -settings were like. This maybe be useful for saving to disk and -feeding to tools like 'grep'. - -@item /changes - -This provides a brief description of the ChangeSource in use -(@pxref{Change Sources}). - -@item /changes/NN - -This shows detailed information about the numbered Change: who was the -author, what files were changed, what revision number was represented, -etc. - -@item /buildslaves - -This summarizes each BuildSlave, including which Builders are -configured to use it, whether the buildslave is currently connected or -not, and host information retrieved from the buildslave itself. - -@item /one_line_per_build - -This page shows one line of text for each build, merging information -from all Builders@footnote{Apparently this is the same way -http://buildd.debian.org displays build status}. Each line specifies -the name of the Builder, the number of the Build, what revision it -used, and a summary of the results. Successful builds are in green, -while failing builds are in red. The date and time of the build are -added to the right-hand edge of the line. The lines are ordered by -build finish timestamp. - -One or more @code{builder=} or @code{branch=} arguments can be used to -restrict the list. In addition, a @code{numbuilds=} argument will -control how many lines are displayed (20 by default). - -@item /builders - -This page shows a small table, with one box for each Builder, -containing the results of the most recent Build. It does not show the -individual steps, or the current status. This is a simple summary of -buildbot status: if this page is green, then all tests are passing. - -As with @code{/one_line_per_build}, this page will also honor -@code{builder=} and @code{branch=} arguments. - -@item /about - -This page gives a brief summary of the Buildbot itself: software -version, versions of some libraries that the Buildbot depends upon, -etc. It also contains a link to the buildbot.net home page. - -@end table - -There are also a set of web-status resources that are intended for use -by other programs, rather than humans. - -@table @code - -@item /xmlrpc - -This runs an XML-RPC server which can be used to query status -information about various builds. See @ref{XMLRPC server} for more -details. - -@end table - -@node WebStatus Configuration Parameters -@subsection WebStatus Configuration Parameters - -@heading HTTP Connection - -The most common way to run a @code{WebStatus} is on a regular TCP -port. To do this, just pass in the TCP port number when you create the -@code{WebStatus} instance; this is called the @code{http_port} argument: - -@example -from buildbot.status.html import WebStatus -c['status'].append(http_port=WebStatus(http_port=8080)) -@end example - -The @code{http_port} argument is actually a ``strports specification'' for the -port that the web server should listen on. This can be a simple port number, or -a string like @code{http_port="tcp:8080:interface=127.0.0.1"} (to limit -connections to the loopback interface, and therefore to clients running on the -same host)@footnote{It may even be possible to provide SSL access by using a -specification like @code{"ssl:12345:privateKey=mykey.pen:certKey=cert.pem"}, -but this is completely untested}. - -If instead (or in addition) you provide the @code{distrib_port} -argument, a twisted.web distributed server will be started either on a -TCP port (if @code{distrib_port} is like @code{"tcp:12345"}) or more -likely on a UNIX socket (if @code{distrib_port} is like -@code{"unix:/path/to/socket"}). - -The @code{public_html} option gives the path to a regular directory of HTML -files that will be displayed alongside the various built-in URLs buildbot -supplies. This is most often used to supply CSS files (@code{/buildbot.css}) -and a top-level navigational file (@code{/index.html}), but can also serve any -other files required - even build results! - -@heading Authorization - -The buildbot web status is, by default, read-only. It displays lots of -information, but users are not allowed to affect the operation of the -buildmaster. However, there are a number of supported activities that can -be enabled, and Buildbot can also perform rudimentary username/password -authentication. The actions are: - -@table @code - -@item forceBuild - -force a particular builder to begin building, optionally with a specific revision, branch, etc. - -@item forceAllBuilds - -force @i{all} builders to start building - -@item pingBuilder - -"ping" a builder's buildslaves to check that they are alive - -@item gracefulShutdown - -gracefully shut down a slave when it is finished with its current build - -@item stopBuild - -stop a running build - -@item stopAllBuilds - -stop all running builds - -@item cancelPendingBuild - -cancel a build that has not yet started - -@end table - -For each of these actions, you can configure buildbot to never allow the -action, always allow the action, allow the action to any authenticated user, or -check with a function of your creation to determine whether the action is OK. - -This is all configured with the @code{Authz} class: - -@example -from buildbot.status.html import WebStatus -from buildbot.status.web.authz import Authz -authz = Authz( - forceBuild=True, - stopBuild=True) -c['status'].append(http_port=WebStatus(http_port=8080, authz=authz)) -@end example - -Each of the actions listed above is an option to @code{Authz}. You can specify -@code{False} (the default) to prohibit that action or @code{True} to enable it. - -@heading Authentication - -If you do not wish to allow strangers to perform actions, but do want -developers to have such access, you will need to add some authentication -support. Pass an instance of @code{status.web.auth.IAuth} as a @code{auth} -keyword argument to @code{Authz}, and specify the action as @code{"auth"}. - -@example -from buildbot.status.html import WebStatus -from buildbot.status.web.authz import Authz -from buildbot.status.web.auth import BasicAuth -users = [('bob', 'secret-pass'), ('jill', 'super-pass')] -authz = Authz(auth=BasicAuth(users), - forceBuild='auth', # only authenticated users - pingBuilder=True, # but anyone can do this -) -c['status'].append(WebStatus(http_port=8080, authz=authz)) -# or -from buildbot.status.web.auth import HTPasswdAuth -auth = (HTPasswdAuth('/path/to/htpasswd')) -@end example - -The class @code{BasicAuth} implements a basic authentication mechanism using a -list of user/password tuples provided from the configuration file. The class -@code{HTPasswdAuth} implements an authentication against an @file{.htpasswd} -file. - -If you need still-more flexibility, pass a function for the authentication -action. That function will be called with an authenticated username and some -action-specific arguments, and should return true if the action is authorized. - -@example -def canForceBuild(username, builder_status): - if builder_status.getName() == 'smoketest': - return True # any authenticated user can run smoketest - elif username == 'releng': - return True # releng can force whatever they want - else: - return False # otherwise, no way. - -authz = Authz(auth=BasicAuth(users), - forceBuild=canForceBuild) -@end example - -The @code{forceBuild} and @code{pingBuilder} actions both supply a -BuilderStatus object. The @code{stopBuild} action supplies a BuildStatus -object. The @code{cancelPendingBuild} action supplies a BuildRequest. The -remainder do not supply any extra arguments. - -@heading Miscellaneous Options - -The @code{changecommentlink} argument can be used to create links to -ticket-ids from change comments (i.e. #123). The argument should -be a tuple of three strings where the first element is a regex that -searches for strings (with match groups), the second is a replace-string -that, when substituted with \1 etc, yields the URL and the third -is the title attribute of the link. -(The @code{} is added by the system.) -So, for Trac tickets (#42, etc): -@code{changecommentlink(r"#(\d+)", r"http://buildbot.net/trac/ticket/\1", r"Ticket \g<0>")} . - -The @code{revlink} is similarly used to create links from revision -ids to a web-view of your source control system. The Use %s to insert -the revision id in the url. I.e. for Buildbot on github: -@code{revlink='http://github.com/djmitche/buildbot/tree/%s'} -(The revision id will be URL encoded before inserted in the replacement string) - -@heading Display-Specific Options - -The @code{order_console_by_time} option affects the rendering of the console; -see the description of the console above. - -The @code{numbuilds} option determines the number of builds that most status -displays will show. It can usually be overriden in the URL, e.g., -@code{?numbuilds=13}. - -The @code{num_events} option gives the default number of events that the -waterfall will display. The @code{num_events_max} gives the maximum number of -events displayed, even if the web browser requests more. - -@node XMLRPC server -@subsection XMLRPC server - -When using WebStatus, the buildbot runs an XML-RPC server at -@file{/xmlrpc} that can be used by other programs to query build -status. The following table lists the methods that can be invoked -using this interface. - -@table @code -@item getAllBuildsInInterval(start, stop) - -Return a list of builds that have completed after the 'start' -timestamp and before the 'stop' timestamp. This looks at all Builders. - -The timestamps are integers, interpreted as standard unix timestamps -(seconds since epoch). - -Each Build is returned as a tuple in the form: @code{(buildername, -buildnumber, build_end, branchname, revision, results, text)} - -The buildnumber is an integer. 'build_end' is an integer (seconds -since epoch) specifying when the build finished. - -The branchname is a string, which may be an empty string to indicate -None (i.e. the default branch). The revision is a string whose meaning -is specific to the VC system in use, and comes from the 'got_revision' -build property. The results are expressed as a string, one of -('success', 'warnings', 'failure', 'exception'). The text is a list of -short strings that ought to be joined by spaces and include slightly -more data about the results of the build. - -@item getBuild(builder_name, build_number) - -Return information about a specific build. - -This returns a dictionary (aka ``struct'' in XMLRPC terms) with -complete information about the build. It does not include the contents -of the log files, but it has just about everything else. - -@end table - -@node MailNotifier, IRC Bot, WebStatus, Status Delivery -@section MailNotifier - -@cindex email -@cindex mail -@stindex buildbot.status.mail.MailNotifier - -The buildbot can also send email when builds finish. The most common -use of this is to tell developers when their change has caused the -build to fail. It is also quite common to send a message to a mailing -list (usually named ``builds'' or similar) about every build. - -The @code{MailNotifier} status target is used to accomplish this. You -configure it by specifying who mail should be sent to, under what -circumstances mail should be sent, and how to deliver the mail. It can -be configured to only send out mail for certain builders, and only -send messages when the build fails, or when the builder transitions -from success to failure. It can also be configured to include various -build logs in each message. - - -By default, the message will be sent to the Interested Users list -(@pxref{Doing Things With Users}), which includes all developers who -made changes in the build. You can add additional recipients with the -extraRecipients argument. - -Each MailNotifier sends mail to a single set of recipients. To send -different kinds of mail to different recipients, use multiple -MailNotifiers. - -The following simple example will send an email upon the completion of -each build, to just those developers whose Changes were included in -the build. The email contains a description of the Build, its results, -and URLs where more information can be obtained. - -@example -from buildbot.status.mail import MailNotifier -mn = MailNotifier(fromaddr="buildbot@@example.org", lookup="example.org") -c['status'].append(mn) -@end example - -To get a simple one-message-per-build (say, for a mailing list), use -the following form instead. This form does not send mail to individual -developers (and thus does not need the @code{lookup=} argument, -explained below), instead it only ever sends mail to the ``extra -recipients'' named in the arguments: - -@example -mn = MailNotifier(fromaddr="buildbot@@example.org", - sendToInterestedUsers=False, - extraRecipients=['listaddr@@example.org']) -@end example - -In some cases it is desirable to have different information then what is -provided in a standard MailNotifier message. For this purpose MailNotifier -provides the argument @code{messageFormatter} (a function) which allows for the -creation of messages with unique content. - -For example, if only short emails are desired (e.g., for delivery to phones) - -@example -from buildbot.status.builder import Results -def messageFormatter(mode, name, build, results, master_status): - result = Results[results] - - text = list() - text.append("STATUS: %s" % result.title()) - return @{ - 'body' : "\n".join(text), - 'type' : 'plain' - @} - -mn = MailNotifier(fromaddr="buildbot@@example.org", - sendToInterestedUsers=False, - mode='problem', - extraRecipients=['listaddr@@example.org'], - messageFormatter=messageFormatter) -@end example - -Another example of a function delivering a customized html email -containing the last 80 lines of logs of a failing build step is given -below: - -@example -from buildbot.status.builder import Results - -def message_formatter(mode, name, build, results, master_status): - """Provide a customized message to BuildBot's MailNotifier. - - The last 80 lines of the log are provided as well as the changes - relevant to the build. Message content is formatted as html. - """ - result = Results[results] - - limit_lines = 80 - text = list() - text.append('

Build status: %s

' % result.upper()) - text.append("Buildslave for this Build: %s" % build.getSlavename()) - text.append('
') - if master_status.getURLForThing(build): - text.append('Complete logs for all build steps: %s' - % (master_status.getURLForThing(build), - master_status.getURLForThing(build)) - ) - text.append('
') - text.append("Build Reason: %s" % build.getReason()) - text.append('
') - - source = "" - ss = build.getSourceStamp() - if ss.branch: - source += "[branch %s] " % ss.branch - if ss.revision: - source += ss.revision - else: - source += "HEAD" - if ss.patch: - source += " (plus patch)" - text.append("Build Source Stamp: %s" % source) - text.append('
') - text.append("Blamelist: %s" % ",".join(build.getResponsibleUsers())) - if ss.changes: - text.append('

Recent Changes:

') - text.extend([c.asHTML() for c in ss.changes]) - - logs = list() - for log in build.getLogs(): - log_name = "%s.%s" % (log.getStep().getName(), log.getName()) - log_status, dummy = log.getStep().getResults() - log_body = log.getText().splitlines() # Note: can be VERY LARGE - log_url = '%s/steps/%s/logs/%s' % (master_status.getURLForThing(build), - log.getStep().getName(), - log.getName()) - logs.append((log_name, log_url, log_body, log_status)) - - name, url, content, logstatus = logs[-1] - - text.append('Detailed log of last build step: %s' - % (url, url)) - text.append('
') - text.append('

Last %d lines of "%s":

' % (limit_lines, name)) - text.append('

') - text.append('
'.join([line for line in - content[len(content)-limit_lines:]])) - text.append('

') - text.append('

') - text.append('-The BuildBot') - return @{ - 'body': "\n".join(text), - 'type': 'html' - @} - -mn = MailNotifier(fromaddr="buildbot@@example.org", - sendToInterestedUsers=False, - mode='failing', - extraRecipients=['listaddr@@example.org'], - messageFormatter=message_formatter) -@end example - - -@heading MailNotifier arguments - -@table @code -@item fromaddr -The email address to be used in the 'From' header. - -@item sendToInterestedUsers -(boolean). If True (the default), send mail to all of the Interested -Users. If False, only send mail to the extraRecipients list. - -@item extraRecipients -(list of strings). A list of email addresses to which messages should -be sent (in addition to the InterestedUsers list, which includes any -developers who made Changes that went into this build). It is a good -idea to create a small mailing list and deliver to that, then let -subscribers come and go as they please. - -@item subject -(string). A string to be used as the subject line of the message. -@code{%(builder)s} will be replaced with the name of the builder which -provoked the message. - -@item mode -(string). Default to 'all'. One of: -@table @code -@item all -Send mail about all builds, bothpassing and failing -@item failing -Only send mail about builds which fail -@item problem -Only send mail about a build which failed when the previous build has passed. -If your builds usually pass, then this will only send mail when a problem -occurs. -@end table - -@item builders -(list of strings). A list of builder names for which mail should be -sent. Defaults to None (send mail for all builds). Use either builders -or categories, but not both. - -@item categories -(list of strings). A list of category names to serve status -information for. Defaults to None (all categories). Use either -builders or categories, but not both. - -@item addLogs -(boolean). If True, include all build logs as attachments to the -messages. These can be quite large. This can also be set to a list of -log names, to send a subset of the logs. Defaults to False. - -@item addPatch -(boolean). If True, include the patch content if a patch was present. -Patches are usually used on a Try server. -Defaults to True. - -@item relayhost -(string). The host to which the outbound SMTP connection should be -made. Defaults to 'localhost' - -@item lookup -(implementor of @code{IEmailLookup}). Object which provides -IEmailLookup, which is responsible for mapping User names (which come -from the VC system) into valid email addresses. If not provided, the -notifier will only be able to send mail to the addresses in the -extraRecipients list. Most of the time you can use a simple Domain -instance. As a shortcut, you can pass as string: this will be treated -as if you had provided Domain(str). For example, -lookup='twistedmatrix.com' will allow mail to be sent to all -developers whose SVN usernames match their twistedmatrix.com account -names. See buildbot/status/mail.py for more details. - -@item messageFormatter - -This is a optional function that can be used to generate a custom mail message. -A @code{messageFormatter} function takes the mail mode (@code{mode}), builder -name (@code{name}), the build status (@code{build}), the result code -(@code{results}), and the BuildMaster status (@code{master_status}). It -returns a dictionary. The @code{body} key gives a string that is the complete -text of the message. The @code{type} key is the message type ('plain' or -'html'). The 'html' type should be used when generating an HTML message. The -@code{subject} key is optional, but gives the subject for the email. -@item extraHeaders -(dictionary) A dictionary containing key/value pairs of extra headers to add -to sent e-mails. Both the keys and the values may be a WithProperties instance. - -@end table - -As a help to those writing @code{messageFormatter} functions, the following -table describes how to get some useful pieces of information from the various -status objects: - -@table @bullet -@item Name of the builder that generated this event -@code{name} -@item Name of the project -@code{master_status.getProjectName()} -@item MailNotifier mode -@code{mode} (one of @code{all, failing, problem, change, passing}) -@item Builder result as a string -@example -from buildbot.status.builder import Results -result_str = Results[results] -# one of 'success', 'warnings', 'failure', 'skipped', or 'exception' -@end example -@item URL to build page -@code{master_status.getURLForThing(build)} -@item URL to buildbot main page. -@code{master_status.getBuildbotURL()} -@item Build text -@code{build.getText()} -@item Mapping of property names to values -@code{build.getProperties()} (a @code{Properties} instance) -@item Slave name -@code{build.getSlavename()} -@item Build reason (from a forced build) -@code{build.getReason()} -@item List of responsible users -@code{build.getResponsibleUsers()} -@item Source information (only valid if ss is not None) -@example -ss = build.getSourceStamp() -if ss: - branch = ss.branch - revision = ss.revision - patch = ss.patch - changes = ss.changes # list -@end example -A change object has the following useful information: -@table @code -@item who -(str) who made this change -@item revision -(str) what VC revision is this change -@item branch -(str) on what branch did this change occur -@item when -(str) when did this change occur -@item files -(list of str) what files were affected in this change -@item comments -(str) comments reguarding the change. -@end table -The @code{Change} methods asText and asHTML return a list of strings with -the above information formatted. -@item Log information -@example -logs = list() -for log in build.getLogs(): - log_name = "%s.%s" % (log.getStep().getName(), log.getName()) - log_status, dummy = log.getStep().getResults() - log_body = log.getText().splitlines() # Note: can be VERY LARGE - log_url = '%s/steps/%s/logs/%s' % (master_status.getURLForThing(build), - log.getStep().getName(), - log.getName()) - logs.append((log_name, log_url, log_body, log_status)) -@end example -@end table - -@node IRC Bot, PBListener, MailNotifier, Status Delivery -@section IRC Bot - -@cindex IRC -@stindex buildbot.status.words.IRC - - -The @code{buildbot.status.words.IRC} status target creates an IRC bot -which will attach to certain channels and be available for status -queries. It can also be asked to announce builds as they occur, or be -told to shut up. - -@example -from buildbot.status import words -irc = words.IRC("irc.example.org", "botnickname", - channels=["channel1", "channel2"], - password="mysecretpassword", - notify_events=@{ - 'exception': 1, - 'successToFailure': 1, - 'failureToSuccess': 1, - @}) -c['status'].append(irc) -@end example - -Take a look at the docstring for @code{words.IRC} for more details on -configuring this service. Note that the @code{useSSL} option requires -@code{PyOpenSSL}. The @code{password} argument, if provided, will be sent to -Nickserv to claim the nickname: some IRC servers will not allow clients to send -private messages until they have logged in with a password. - -To use the service, you address messages at the buildbot, either -normally (@code{botnickname: status}) or with private messages -(@code{/msg botnickname status}). The buildbot will respond in kind. - -Some of the commands currently available: - -@table @code - -@item list builders -Emit a list of all configured builders -@item status BUILDER -Announce the status of a specific Builder: what it is doing right now. -@item status all -Announce the status of all Builders -@item watch BUILDER -If the given Builder is currently running, wait until the Build is -finished and then announce the results. -@item last BUILDER -Return the results of the last build to run on the given Builder. -@item join CHANNEL -Join the given IRC channel -@item leave CHANNEL -Leave the given IRC channel -@item notify on|off|list EVENT -Report events relating to builds. If the command is issued as a -private message, then the report will be sent back as a private -message to the user who issued the command. Otherwise, the report -will be sent to the channel. Available events to be notified are: - -@table @code -@item started -A build has started -@item finished -A build has finished -@item success -A build finished successfully -@item failed -A build failed -@item exception -A build generated and exception -@item xToY -The previous build was x, but this one is Y, where x and Y are each -one of success, warnings, failure, exception (except Y is -capitalized). For example: successToFailure will notify if the -previous build was successful, but this one failed -@end table - -@item help COMMAND -Describe a command. Use @code{help commands} to get a list of known -commands. -@item source -Announce the URL of the Buildbot's home page. -@item version -Announce the version of this Buildbot. -@end table - -Additionally, the config file may specify default notification options -as shown in the example earlier. - -If the @code{allowForce=True} option was used, some addtional commands -will be available: - -@table @code -@item force build BUILDER REASON -Tell the given Builder to start a build of the latest code. The user -requesting the build and REASON are recorded in the Build status. The -buildbot will announce the build's status when it finishes. - -@item stop build BUILDER REASON -Terminate any running build in the given Builder. REASON will be added -to the build status to explain why it was stopped. You might use this -if you committed a bug, corrected it right away, and don't want to -wait for the first build (which is destined to fail) to complete -before starting the second (hopefully fixed) build. -@end table - -@node PBListener, Writing New Status Plugins, IRC Bot, Status Delivery -@section PBListener - -@cindex PBListener -@stindex buildbot.status.client.PBListener - - -@example -import buildbot.status.client -pbl = buildbot.status.client.PBListener(port=int, user=str, - passwd=str) -c['status'].append(pbl) -@end example - -This sets up a PB listener on the given TCP port, to which a PB-based -status client can connect and retrieve status information. -@code{buildbot statusgui} (@pxref{statusgui}) is an example of such a -status client. The @code{port} argument can also be a strports -specification string. - -@node Writing New Status Plugins, , PBListener, Status Delivery -@section Writing New Status Plugins - -TODO: this needs a lot more examples - -Each status plugin is an object which provides the -@code{twisted.application.service.IService} interface, which creates a -tree of Services with the buildmaster at the top [not strictly true]. -The status plugins are all children of an object which implements -@code{buildbot.interfaces.IStatus}, the main status object. From this -object, the plugin can retrieve anything it wants about current and -past builds. It can also subscribe to hear about new and upcoming -builds. - -Status plugins which only react to human queries (like the Waterfall -display) never need to subscribe to anything: they are idle until -someone asks a question, then wake up and extract the information they -need to answer it, then they go back to sleep. Plugins which need to -act spontaneously when builds complete (like the MailNotifier plugin) -need to subscribe to hear about new builds. - -If the status plugin needs to run network services (like the HTTP -server used by the Waterfall plugin), they can be attached as Service -children of the plugin itself, using the @code{IServiceCollection} -interface. - - - -@node Command-line tool, Resources, Status Delivery, Top -@chapter Command-line tool - -The @command{buildbot} command-line tool can be used to start or stop a -buildmaster or buildbot, and to interact with a running buildmaster. -Some of its subcommands are intended for buildmaster admins, while -some are for developers who are editing the code that the buildbot is -monitoring. - -@menu -* Administrator Tools:: -* Developer Tools:: -* Other Tools:: -* .buildbot config directory:: -@end menu - -@node Administrator Tools, Developer Tools, Command-line tool, Command-line tool -@section Administrator Tools - -The following @command{buildbot} sub-commands are intended for -buildmaster administrators: - -@heading create-master - -This creates a new directory and populates it with files that allow it -to be used as a buildmaster's base directory. - -@example -buildbot create-master BASEDIR -@end example - -@heading create-slave - -This creates a new directory and populates it with files that let it -be used as a buildslave's base directory. You must provide several -arguments, which are used to create the initial @file{buildbot.tac} -file. - -@example -buildbot create-slave @var{BASEDIR} @var{MASTERHOST}:@var{PORT} @var{SLAVENAME} @var{PASSWORD} -@end example - -@heading start - -This starts a buildmaster or buildslave which was already created in -the given base directory. The daemon is launched in the background, -with events logged to a file named @file{twistd.log}. - -@example -buildbot start BASEDIR -@end example - -@heading stop - -This terminates the daemon (either buildmaster or buildslave) running -in the given directory. - -@example -buildbot stop BASEDIR -@end example - -@heading sighup - -This sends a SIGHUP to the buildmaster running in the given directory, -which causes it to re-read its @file{master.cfg} file. - -@example -buildbot sighup BASEDIR -@end example - -@node Developer Tools, Other Tools, Administrator Tools, Command-line tool -@section Developer Tools - -These tools are provided for use by the developers who are working on -the code that the buildbot is monitoring. - -@menu -* statuslog:: -* statusgui:: -* try:: -@end menu - -@node statuslog, statusgui, Developer Tools, Developer Tools -@subsection statuslog - -@example -buildbot statuslog --master @var{MASTERHOST}:@var{PORT} -@end example - -This command starts a simple text-based status client, one which just -prints out a new line each time an event occurs on the buildmaster. - -The @option{--master} option provides the location of the -@code{buildbot.status.client.PBListener} status port, used to deliver -build information to realtime status clients. The option is always in -the form of a string, with hostname and port number separated by a -colon (@code{HOSTNAME:PORTNUM}). Note that this port is @emph{not} the -same as the slaveport (although a future version may allow the same -port number to be used for both purposes). If you get an error message -to the effect of ``Failure: twisted.cred.error.UnauthorizedLogin:'', -this may indicate that you are connecting to the slaveport rather than -a @code{PBListener} port. - -The @option{--master} option can also be provided by the -@code{masterstatus} name in @file{.buildbot/options} (@pxref{.buildbot -config directory}). - -@node statusgui, try, statuslog, Developer Tools -@subsection statusgui - -@cindex statusgui - -If you have set up a PBListener (@pxref{PBListener}), you will be able -to monitor your Buildbot using a simple Gtk+ application invoked with -the @code{buildbot statusgui} command: - -@example -buildbot statusgui --master @var{MASTERHOST}:@var{PORT} -@end example - -This command starts a simple Gtk+-based status client, which contains a few -boxes for each Builder that change color as events occur. It uses the same -@option{--master} argument and @code{masterstatus} option as the -@command{buildbot statuslog} command (@pxref{statuslog}). - -@node try, , statusgui, Developer Tools -@subsection try - -This lets a developer to ask the question ``What would happen if I -committed this patch right now?''. It runs the unit test suite (across -multiple build platforms) on the developer's current code, allowing -them to make sure they will not break the tree when they finally -commit their changes. - -The @command{buildbot try} command is meant to be run from within a -developer's local tree, and starts by figuring out the base revision -of that tree (what revision was current the last time the tree was -updated), and a patch that can be applied to that revision of the tree -to make it match the developer's copy. This (revision, patch) pair is -then sent to the buildmaster, which runs a build with that -SourceStamp. If you want, the tool will emit status messages as the -builds run, and will not terminate until the first failure has been -detected (or the last success). - -There is an alternate form which accepts a pre-made patch file -(typically the output of a command like 'svn diff'). This ``--diff'' -form does not require a local tree to run from. See @xref{try --diff}. - -For this command to work, several pieces must be in place: - - -@heading TryScheduler - -@slindex buildbot.scheduler.Try_Jobdir -@slindex buildbot.scheduler.Try_Userpass - -The buildmaster must have a @code{scheduler.Try} instance in -the config file's @code{c['schedulers']} list. This lets the -administrator control who may initiate these ``trial'' builds, which -branches are eligible for trial builds, and which Builders should be -used for them. - -The @code{TryScheduler} has various means to accept build requests: -all of them enforce more security than the usual buildmaster ports do. -Any source code being built can be used to compromise the buildslave -accounts, but in general that code must be checked out from the VC -repository first, so only people with commit privileges can get -control of the buildslaves. The usual force-build control channels can -waste buildslave time but do not allow arbitrary commands to be -executed by people who don't have those commit privileges. However, -the source code patch that is provided with the trial build does not -have to go through the VC system first, so it is important to make -sure these builds cannot be abused by a non-committer to acquire as -much control over the buildslaves as a committer has. Ideally, only -developers who have commit access to the VC repository would be able -to start trial builds, but unfortunately the buildmaster does not, in -general, have access to VC system's user list. - -As a result, the @code{TryScheduler} requires a bit more -configuration. There are currently two ways to set this up: - -@table @strong -@item jobdir (ssh) - -This approach creates a command queue directory, called the -``jobdir'', in the buildmaster's working directory. The buildmaster -admin sets the ownership and permissions of this directory to only -grant write access to the desired set of developers, all of whom must -have accounts on the machine. The @code{buildbot try} command creates -a special file containing the source stamp information and drops it in -the jobdir, just like a standard maildir. When the buildmaster notices -the new file, it unpacks the information inside and starts the builds. - -The config file entries used by 'buildbot try' either specify a local -queuedir (for which write and mv are used) or a remote one (using scp -and ssh). - -The advantage of this scheme is that it is quite secure, the -disadvantage is that it requires fiddling outside the buildmaster -config (to set the permissions on the jobdir correctly). If the -buildmaster machine happens to also house the VC repository, then it -can be fairly easy to keep the VC userlist in sync with the -trial-build userlist. If they are on different machines, this will be -much more of a hassle. It may also involve granting developer accounts -on a machine that would not otherwise require them. - -To implement this, the buildslave invokes 'ssh -l username host -buildbot tryserver ARGS', passing the patch contents over stdin. The -arguments must include the inlet directory and the revision -information. - -@item user+password (PB) - -In this approach, each developer gets a username/password pair, which -are all listed in the buildmaster's configuration file. When the -developer runs @code{buildbot try}, their machine connects to the -buildmaster via PB and authenticates themselves using that username -and password, then sends a PB command to start the trial build. - -The advantage of this scheme is that the entire configuration is -performed inside the buildmaster's config file. The disadvantages are -that it is less secure (while the ``cred'' authentication system does -not expose the password in plaintext over the wire, it does not offer -most of the other security properties that SSH does). In addition, the -buildmaster admin is responsible for maintaining the username/password -list, adding and deleting entries as developers come and go. - -@end table - - -For example, to set up the ``jobdir'' style of trial build, using a -command queue directory of @file{MASTERDIR/jobdir} (and assuming that -all your project developers were members of the @code{developers} unix -group), you would first create that directory (with @command{mkdir -MASTERDIR/jobdir MASTERDIR/jobdir/new MASTERDIR/jobdir/cur -MASTERDIR/jobdir/tmp; chgrp developers MASTERDIR/jobdir -MASTERDIR/jobdir/*; chmod g+rwx,o-rwx MASTERDIR/jobdir -MASTERDIR/jobdir/*}), and then use the following scheduler in the -buildmaster's config file: - -@example -from buildbot.scheduler import Try_Jobdir -s = Try_Jobdir("try1", ["full-linux", "full-netbsd", "full-OSX"], - jobdir="jobdir") -c['schedulers'] = [s] -@end example - -Note that you must create the jobdir before telling the buildmaster to -use this configuration, otherwise you will get an error. Also remember -that the buildmaster must be able to read and write to the jobdir as -well. Be sure to watch the @file{twistd.log} file (@pxref{Logfiles}) -as you start using the jobdir, to make sure the buildmaster is happy -with it. - -To use the username/password form of authentication, create a -@code{Try_Userpass} instance instead. It takes the same -@code{builderNames} argument as the @code{Try_Jobdir} form, but -accepts an addtional @code{port} argument (to specify the TCP port to -listen on) and a @code{userpass} list of username/password pairs to -accept. Remember to use good passwords for this: the security of the -buildslave accounts depends upon it: - -@example -from buildbot.scheduler import Try_Userpass -s = Try_Userpass("try2", ["full-linux", "full-netbsd", "full-OSX"], - port=8031, userpass=[("alice","pw1"), ("bob", "pw2")] ) -c['schedulers'] = [s] -@end example - -Like most places in the buildbot, the @code{port} argument takes a -strports specification. See @code{twisted.application.strports} for -details. - - -@heading locating the master - -The @command{try} command needs to be told how to connect to the -@code{TryScheduler}, and must know which of the authentication -approaches described above is in use by the buildmaster. You specify -the approach by using @option{--connect=ssh} or @option{--connect=pb} -(or @code{try_connect = 'ssh'} or @code{try_connect = 'pb'} in -@file{.buildbot/options}). - -For the PB approach, the command must be given a @option{--master} -argument (in the form HOST:PORT) that points to TCP port that you -picked in the @code{Try_Userpass} scheduler. It also takes a -@option{--username} and @option{--passwd} pair of arguments that match -one of the entries in the buildmaster's @code{userpass} list. These -arguments can also be provided as @code{try_master}, -@code{try_username}, and @code{try_password} entries in the -@file{.buildbot/options} file. - -For the SSH approach, the command must be given @option{--tryhost}, -@option{--username}, and optionally @option{--password} (TODO: -really?) to get to the buildmaster host. It must also be given -@option{--trydir}, which points to the inlet directory configured -above. The trydir can be relative to the user's home directory, but -most of the time you will use an explicit path like -@file{~buildbot/project/trydir}. These arguments can be provided in -@file{.buildbot/options} as @code{try_host}, @code{try_username}, -@code{try_password}, and @code{try_dir}. - -In addition, the SSH approach needs to connect to a PBListener status -port, so it can retrieve and report the results of the build (the PB -approach uses the existing connection to retrieve status information, -so this step is not necessary). This requires a @option{--master} -argument, or a @code{masterstatus} entry in @file{.buildbot/options}, -in the form of a HOSTNAME:PORT string. - - -@heading choosing the Builders - -A trial build is performed on multiple Builders at the same time, and -the developer gets to choose which Builders are used (limited to a set -selected by the buildmaster admin with the TryScheduler's -@code{builderNames=} argument). The set you choose will depend upon -what your goals are: if you are concerned about cross-platform -compatibility, you should use multiple Builders, one from each -platform of interest. You might use just one builder if that platform -has libraries or other facilities that allow better test coverage than -what you can accomplish on your own machine, or faster test runs. - -The set of Builders to use can be specified with multiple -@option{--builder} arguments on the command line. It can also be -specified with a single @code{try_builders} option in -@file{.buildbot/options} that uses a list of strings to specify all -the Builder names: - -@example -try_builders = ["full-OSX", "full-win32", "full-linux"] -@end example - -If you are using the PB approach, you can get the names of the builders -that are configured for the try scheduler using the @code{get-builder-names} -argument: - -@example -buildbot try --get-builder-names --connect=pb --master=... --username=... --passwd=... -@end example - -@heading specifying the VC system - -The @command{try} command also needs to know how to take the -developer's current tree and extract the (revision, patch) -source-stamp pair. Each VC system uses a different process, so you -start by telling the @command{try} command which VC system you are -using, with an argument like @option{--vc=cvs} or @option{--vc=tla}. -This can also be provided as @code{try_vc} in -@file{.buildbot/options}. - -@c The order of this list comes from the end of scripts/tryclient.py - -The following names are recognized: @code{cvs} @code{svn} @code{baz} -@code{tla} @code{bzr} @code{hg} @code{darcs} @code{git} @code{p4} - - -@heading finding the top of the tree - -Some VC systems (notably CVS and SVN) track each directory -more-or-less independently, which means the @command{try} command -needs to move up to the top of the project tree before it will be able -to construct a proper full-tree patch. To accomplish this, the -@command{try} command will crawl up through the parent directories -until it finds a marker file. The default name for this marker file is -@file{.buildbot-top}, so when you are using CVS or SVN you should -@code{touch .buildbot-top} from the top of your tree before running -@command{buildbot try}. Alternatively, you can use a filename like -@file{ChangeLog} or @file{README}, since many projects put one of -these files in their top-most directory (and nowhere else). To set -this filename, use @option{--try-topfile=ChangeLog}, or set it in the -options file with @code{try_topfile = 'ChangeLog'}. - -You can also manually set the top of the tree with -@option{--try-topdir=~/trees/mytree}, or @code{try_topdir = -'~/trees/mytree'}. If you use @code{try_topdir}, in a -@file{.buildbot/options} file, you will need a separate options file -for each tree you use, so it may be more convenient to use the -@code{try_topfile} approach instead. - -Other VC systems which work on full projects instead of individual -directories (tla, baz, darcs, monotone, mercurial, git) do not require -@command{try} to know the top directory, so the @option{--try-topfile} -and @option{--try-topdir} arguments will be ignored. -@c is this true? I think I currently require topdirs all the time. - -If the @command{try} command cannot find the top directory, it will -abort with an error message. - -@heading determining the branch name - -Some VC systems record the branch information in a way that ``try'' -can locate it, in particular Arch (both @command{tla} and -@command{baz}). For the others, if you are using something other than -the default branch, you will have to tell the buildbot which branch -your tree is using. You can do this with either the @option{--branch} -argument, or a @option{try_branch} entry in the -@file{.buildbot/options} file. - -@heading determining the revision and patch - -Each VC system has a separate approach for determining the tree's base -revision and computing a patch. - -@table @code - -@item CVS - -@command{try} pretends that the tree is up to date. It converts the -current time into a @code{-D} time specification, uses it as the base -revision, and computes the diff between the upstream tree as of that -point in time versus the current contents. This works, more or less, -but requires that the local clock be in reasonably good sync with the -repository. - -@item SVN -@command{try} does a @code{svn status -u} to find the latest -repository revision number (emitted on the last line in the ``Status -against revision: NN'' message). It then performs an @code{svn diff --rNN} to find out how your tree differs from the repository version, -and sends the resulting patch to the buildmaster. If your tree is not -up to date, this will result in the ``try'' tree being created with -the latest revision, then @emph{backwards} patches applied to bring it -``back'' to the version you actually checked out (plus your actual -code changes), but this will still result in the correct tree being -used for the build. - -@item baz -@command{try} does a @code{baz tree-id} to determine the -fully-qualified version and patch identifier for the tree -(ARCHIVE/VERSION--patch-NN), and uses the VERSION--patch-NN component -as the base revision. It then does a @code{baz diff} to obtain the -patch. - -@item tla -@command{try} does a @code{tla tree-version} to get the -fully-qualified version identifier (ARCHIVE/VERSION), then takes the -first line of @code{tla logs --reverse} to figure out the base -revision. Then it does @code{tla changes --diffs} to obtain the patch. - -@item bzr -@command{try} does a @code{bzr revision-info} to find the base revision, -then a @code{bzr diff -r$base..} to obtain the patch. - -@item Mercurial -@code{hg identify --debug} emits the full revision id (as opposed to -the common 12-char truncated) which is a SHA1 hash of the current -revision's contents. This is used as the base revision. -@code{hg diff} then provides the patch relative to that -revision. For @command{try} to work, your working directory must only -have patches that are available from the same remotely-available -repository that the build process' @code{source.Mercurial} will use. - -@item Perforce -@command{try} does a @code{p4 changes -m1 ...} to determine the latest -changelist and implicitly assumes that the local tree is synched to this -revision. This is followed by a @code{p4 diff -du} to obtain the patch. -A p4 patch differs sligtly from a normal diff. It contains full depot -paths and must be converted to paths relative to the branch top. To convert -the following restriction is imposed. The p4base (see @pxref{P4Source}) - is assumed to be @code{//depot} - -@item Darcs -@command{try} does a @code{darcs changes --context} to find the list -of all patches back to and including the last tag that was made. This text -file (plus the location of a repository that contains all these -patches) is sufficient to re-create the tree. Therefore the contents -of this ``context'' file @emph{are} the revision stamp for a -Darcs-controlled source tree. It then does a @code{darcs diff --u} to compute the patch relative to that revision. - -@item Git -@code{git branch -v} lists all the branches available in the local -repository along with the revision ID it points to and a short summary -of the last commit. The line containing the currently checked out -branch begins with '* ' (star and space) while all the others start -with ' ' (two spaces). @command{try} scans for this line and extracts -the branch name and revision from it. Then it generates a diff against -the base revision. -@c TODO: I'm not sure if this actually works the way it's intended -@c since the extracted base revision might not actually exist in the -@c upstream repository. Perhaps we need to add a --remote option to -@c specify the remote tracking branch to generate a diff against. - -@c TODO: monotone -@end table - -@heading waiting for results - -If you provide the @option{--wait} option (or @code{try_wait = True} -in @file{.buildbot/options}), the @command{buildbot try} command will -wait until your changes have either been proven good or bad before -exiting. Unless you use the @option{--quiet} option (or -@code{try_quiet=True}), it will emit a progress message every 60 -seconds until the builds have completed. - -@menu -* try --diff:: -@end menu - -@node try --diff, , try, try -@subsubsection try --diff - -Sometimes you might have a patch from someone else that you want to -submit to the buildbot. For example, a user may have created a patch -to fix some specific bug and sent it to you by email. You've inspected -the patch and suspect that it might do the job (and have at least -confirmed that it doesn't do anything evil). Now you want to test it -out. - -One approach would be to check out a new local tree, apply the patch, -run your local tests, then use ``buildbot try'' to run the tests on -other platforms. An alternate approach is to use the @command{buildbot -try --diff} form to have the buildbot test the patch without using a -local tree. - -This form takes a @option{--diff} argument which points to a file that -contains the patch you want to apply. By default this patch will be -applied to the TRUNK revision, but if you give the optional -@option{--baserev} argument, a tree of the given revision will be used -as a starting point instead of TRUNK. - -You can also use @command{buildbot try --diff=-} to read the patch -from stdin. - -Each patch has a ``patchlevel'' associated with it. This indicates the -number of slashes (and preceding pathnames) that should be stripped -before applying the diff. This exactly corresponds to the @option{-p} -or @option{--strip} argument to the @command{patch} utility. By -default @command{buildbot try --diff} uses a patchlevel of 0, but you -can override this with the @option{-p} argument. - -When you use @option{--diff}, you do not need to use any of the other -options that relate to a local tree, specifically @option{--vc}, -@option{--try-topfile}, or @option{--try-topdir}. These options will -be ignored. Of course you must still specify how to get to the -buildmaster (with @option{--connect}, @option{--tryhost}, etc). - - -@node Other Tools, .buildbot config directory, Developer Tools, Command-line tool -@section Other Tools - -These tools are generally used by buildmaster administrators. - -@menu -* sendchange:: -* debugclient:: +* Index of Useful Classes:: +* Index of master.cfg keys:: +* Index:: Complete index. @end menu -@node sendchange, debugclient, Other Tools, Other Tools -@subsection sendchange - -This command is used to tell the buildmaster about source changes. It -is intended to be used from within a commit script, installed on the -VC server. It requires that you have a PBChangeSource -(@pxref{PBChangeSource}) running in the buildmaster (by being set in -@code{c['change_source']}). - - -@example -buildbot sendchange --master @var{MASTERHOST}:@var{PORT} --username @var{USER} @var{FILENAMES..} -@end example - -The @code{master} and @code{username} arguments can also be given in the -options file (@pxref{.buildbot config directory}). There are other (optional) -arguments which can influence the @code{Change} that gets submitted: - -@table @code -@item --branch -(or option @code{branch}) This provides the (string) branch specifier. If -omitted, it defaults to None, indicating the ``default branch''. All files -included in this Change must be on the same branch. - -@item --category -(or option @code{category}) This provides the (string) category specifier. If -omitted, it defaults to None, indicating ``no category''. The category property -is used by Schedulers to filter what changes they listen to. - -@item --revision_number -This provides a (numeric) revision number for the change, used for VC systems -that use numeric transaction numbers (like Subversion). - -@item --revision -This provides a (string) revision specifier, for VC systems that use -strings (Arch would use something like patch-42 etc). - -@item --revision_file -This provides a filename which will be opened and the contents used as -the revision specifier. This is specifically for Darcs, which uses the -output of @command{darcs changes --context} as a revision specifier. -This context file can be a couple of kilobytes long, spanning a couple -lines per patch, and would be a hassle to pass as a command-line -argument. - -@item --property -This parameter is used to set a property on the Change generated by sendchange. -Properties are specified as a name:value pair, separated by a colon. You may -specify many properties by passing this parameter multiple times. - -@item --comments -This provides the change comments as a single argument. You may want -to use @option{--logfile} instead. - -@item --logfile -This instructs the tool to read the change comments from the given -file. If you use @code{-} as the filename, the tool will read the -change comments from stdin. -@end table - - -@node debugclient, , sendchange, Other Tools -@subsection debugclient - -@example -buildbot debugclient --master @var{MASTERHOST}:@var{PORT} --passwd @var{DEBUGPW} -@end example - -This launches a small Gtk+/Glade-based debug tool, connecting to the -buildmaster's ``debug port''. This debug port shares the same port -number as the slaveport (@pxref{Setting the slaveport}), but the -@code{debugPort} is only enabled if you set a debug password in the -buildmaster's config file (@pxref{Debug options}). The -@option{--passwd} option must match the @code{c['debugPassword']} -value. - -@option{--master} can also be provided in @file{.debug/options} by the -@code{master} key. @option{--passwd} can be provided by the -@code{debugPassword} key. @xref{.buildbot config directory}. - -The @code{Connect} button must be pressed before any of the other -buttons will be active. This establishes the connection to the -buildmaster. The other sections of the tool are as follows: - -@table @code -@item Reload .cfg -Forces the buildmaster to reload its @file{master.cfg} file. This is -equivalent to sending a SIGHUP to the buildmaster, but can be done -remotely through the debug port. Note that it is a good idea to be -watching the buildmaster's @file{twistd.log} as you reload the config -file, as any errors which are detected in the config file will be -announced there. - -@item Rebuild .py -(not yet implemented). The idea here is to use Twisted's ``rebuild'' -facilities to replace the buildmaster's running code with a new -version. Even if this worked, it would only be used by buildbot -developers. - -@item poke IRC -This locates a @code{words.IRC} status target and causes it to emit a -message on all the channels to which it is currently connected. This -was used to debug a problem in which the buildmaster lost the -connection to the IRC server and did not attempt to reconnect. - -@item Commit -This allows you to inject a Change, just as if a real one had been -delivered by whatever VC hook you are using. You can set the name of -the committed file and the name of the user who is doing the commit. -Optionally, you can also set a revision for the change. If the -revision you provide looks like a number, it will be sent as an -integer, otherwise it will be sent as a string. - -@item Force Build -This lets you force a Builder (selected by name) to start a build of -the current source tree. - -@item Currently -(obsolete). This was used to manually set the status of the given -Builder, but the status-assignment code was changed in an incompatible -way and these buttons are no longer meaningful. - -@end table - - -@node .buildbot config directory, , Other Tools, Command-line tool -@section .buildbot config directory - -Many of the @command{buildbot} tools must be told how to contact the -buildmaster that they interact with. This specification can be -provided as a command-line argument, but most of the time it will be -easier to set them in an ``options'' file. The @command{buildbot} -command will look for a special directory named @file{.buildbot}, -starting from the current directory (where the command was run) and -crawling upwards, eventually looking in the user's home directory. It -will look for a file named @file{options} in this directory, and will -evaluate it as a python script, looking for certain names to be set. -You can just put simple @code{name = 'value'} pairs in this file to -set the options. - -For a description of the names used in this file, please see the -documentation for the individual @command{buildbot} sub-commands. The -following is a brief sample of what this file's contents could be. - -@example -# for status-reading tools -masterstatus = 'buildbot.example.org:12345' -# for 'sendchange' or the debug port -master = 'buildbot.example.org:18990' -debugPassword = 'eiv7Po' -@end example - -Note carefully that the names in the @code{options} file usually do not match -the command-line option name. - -@table @code -@item masterstatus -Equivalent to @code{--master} for @ref{statuslog} and @ref{statusgui}, this -gives the location of the @code{client.PBListener} status port. - -@item master -Equivalent to @code{--master} for @ref{debugclient} and @ref{sendchange}. -This option is used for two purposes. It is the location of the -@code{debugPort} for @command{debugclient} and the location of the -@code{pb.PBChangeSource} for @command{sendchange}. Generally these are the -same port. - -@item debugPassword -Equivalent to @code{--passwd} for @ref{debugclient}. - -XXX Must match the value of @code{c['debugPassword']}, used to protect the -debug port, for the @command{debugclient} command. - -@item username -Equivalent to @code{--username} for the @ref{sendchange} command. - -@item branch -Equivalent to @code{--branch} for the @ref{sendchange} command. - -@item category -Equivalent to @code{--category} for the @ref{sendchange} command. - -@item try_connect -Equivalent to @code{--connect}, this specifies how the @ref{try} command should -deliver its request to the buildmaster. The currently accepted values are -``ssh'' and ``pb''. - -@item try_builders -Equivalent to @code{--builders}, specifies which builders should be used for -the @ref{try} build. - -@item try_vc -Equivalent to @code{--vc} for @ref{try}, this specifies the version control -system being used. - -@item try_branch -Equivlanent to @code{--branch}, this indicates that the current tree is on a non-trunk branch. +@node Introduction +@chapter Introduction +@include introduction.texinfo -@item try_topdir -@item try_topfile -Use @code{try_topdir}, equivalent to @code{--try-topdir}, to explicitly -indicate the top of your working tree, or @code{try_topfile}, equivalent to -@code{--try-topfile} to name a file that will only be found in that top-most -directory. +@node Installation +@chapter Installation +@include installation.texinfo -@item try_host -@item try_username -@item try_dir -When @code{try_connect} is ``ssh'', the command will use @code{try_host} for -@code{--tryhost}, @code{try_username} for @code{--username}, and @code{try_dir} -for @code{--trydir}. Apologies for the confusing presence and absence of -'try'. +@node Concepts +@chapter Concepts +@include concepts.texinfo -@item try_username -@item try_password -@item try_master -Similarly, when @code{try_connect} is ``pb'', the command will pay attention to -@code{try_username} for @code{--username}, @code{try_password} for -@code{--passwd}, and @code{try_master} for @code{--master}. +@node Configuration +@chapter Configuration +@include configuration.texinfo -@item try_wait -@item masterstatus -@code{try_wait} and @code{masterstatus} (equivalent to @code{--wait} and -@code{master}, respectively) are used to ask the @ref{try} command to wait for -the requested build to complete. +@node Getting Source Code Changes +@chapter Getting Source Code Changes +@include changesources.texinfo -@end table +@node Build Process +@chapter Build Process +@include buildprocess.texinfo +@node Status Delivery +@chapter Status Delivery +@include status.texinfo +@node Command-line Tool +@chapter Command-line Tool +@include cmdline.texinfo -@node Resources, Developer's Appendix, Command-line tool, Top +@node Resources @chapter Resources +@include resources.texinfo -The Buildbot home page is @uref{http://buildbot.net/}. - -For configuration questions and general discussion, please use the -@code{buildbot-devel} mailing list. The subscription instructions and -archives are available at -@uref{http://lists.sourceforge.net/lists/listinfo/buildbot-devel} - -The #buildbot channel on Freenode's IRC servers hosts development discussion, and -often folks are available to answer questions there, as well. - -@node Developer's Appendix, Index of Useful Classes, Resources, Top -@unnumbered Developer's Appendix - -This appendix contains random notes about the implementation of the -Buildbot, and is likely to only be of use to people intending to -extend the Buildbot's internals. - -The buildmaster consists of a tree of Service objects, which is shaped -as follows: - -@example -BuildMaster - ChangeMaster (in .change_svc) - [IChangeSource instances] - [IScheduler instances] (in .schedulers) - BotMaster (in .botmaster) - [IBuildSlave instances] - [IStatusTarget instances] (in .statusTargets) -@end example - -The BotMaster has a collection of Builder objects as values of its -@code{.builders} dictionary. - - -@node Index of Useful Classes, Index of master.cfg keys, Developer's Appendix, Top +@node Index of Useful Classes @unnumbered Index of Useful Classes This is a list of all user-visible classes. There are the ones that @@ -9511,7 +109,6 @@ are useful in @file{master.cfg}, the buildmaster's configuration file. Classes that are not listed here are generally internal things that admins are unlikely to have much use for. - @heading Change Sources @printindex cs @@ -9524,24 +121,17 @@ admins are unlikely to have much use for. @heading Build Steps @printindex bs -@c undocumented steps -@bsindex buildbot.steps.source.Git -@bsindex buildbot.steps.maxq.MaxQ - - @heading Status Targets @printindex st -@c TODO: undocumented targets - -@node Index of master.cfg keys, Index, Index of Useful Classes, Top +@node Index of master.cfg keys @unnumbered Index of master.cfg keys -This is a list of all of the significant keys in master.cfg . Recall -that master.cfg is effectively a small python program with exactly one -responsibility: create a dictionary named @code{BuildmasterConfig}. -The keys of this dictionary are listed here. The beginning of the -master.cfg file typically starts with something like: +This is a list of all of the significant keys in @code{master.cfg}. Recall that +@code{master.cfg} is effectively a small python program with exactly one +responsibility: create a dictionary named @code{BuildmasterConfig}. The keys +of this dictionary are listed here. The beginning of the master.cfg file +typically starts with something like: @example BuildmasterConfig = c = @{@} @@ -9553,7 +143,7 @@ master.cfg as @code{c['change_source']}. @printindex bc -@node Index, , Index of master.cfg keys, Top +@node Index @unnumbered Index @printindex cp diff --git a/docs/buildprocess.texinfo b/docs/buildprocess.texinfo new file mode 100644 index 00000000000..68272cdd10e --- /dev/null +++ b/docs/buildprocess.texinfo @@ -0,0 +1,2914 @@ +A @code{Build} object is responsible for actually performing a build. +It gets access to a remote @code{SlaveBuilder} where it may run +commands, and a @code{BuildStatus} object where it must emit status +events. The @code{Build} is created by the Builder's +@code{BuildFactory}. + +The default @code{Build} class is made up of a fixed sequence of +@code{BuildSteps}, executed one after another until all are complete +(or one of them indicates that the build should be halted early). The +default @code{BuildFactory} creates instances of this @code{Build} +class with a list of @code{BuildSteps}, so the basic way to configure +the build is to provide a list of @code{BuildSteps} to your +@code{BuildFactory}. + +More complicated @code{Build} subclasses can make other decisions: +execute some steps only if certain files were changed, or if certain +previous steps passed or failed. The base class has been written to +allow users to express basic control flow without writing code, but +you can always subclass and customize to achieve more specialized +behavior. + +@menu +* Build Steps:: +* Interlocks:: +* Build Factories:: +@end menu + +@node Build Steps +@section Build Steps + +@code{BuildStep}s are usually specified in the buildmaster's +configuration file, in a list that goes into the @code{BuildFactory}. +The @code{BuildStep} instances in this list are used as templates to +construct new independent copies for each build (so that state can be +kept on the @code{BuildStep} in one build without affecting a later +build). Each @code{BuildFactory} can be created with a list of steps, +or the factory can be created empty and then steps added to it using +the @code{addStep} method: + +@example +from buildbot.steps import source, shell +from buildbot.process import factory + +f = factory.BuildFactory() +f.addStep(source.SVN(svnurl="http://svn.example.org/Trunk/")) +f.addStep(shell.ShellCommand(command=["make", "all"])) +f.addStep(shell.ShellCommand(command=["make", "test"])) +@end example + +In earlier versions (0.7.5 and older), these steps were specified with +a tuple of (step_class, keyword_arguments). Steps can still be +specified this way, but the preferred form is to pass actual +@code{BuildStep} instances to @code{addStep}, because that gives the +@code{BuildStep} class a chance to do some validation on the +arguments. + +If you have a common set of steps which are used in several factories, the +@code{addSteps} method may be handy. It takes an iterable of @code{BuildStep} +instances. + +@example +setup_steps = [ + source.SVN(svnurl="http://svn.example.org/Trunk/") + shell.ShellCommand(command="./setup") +] +quick = factory.BuildFactory() +quick.addSteps(setup_steps) +quick.addStep(shell.shellCommand(command="make quick")) +@end example + +The rest of this section lists all the standard BuildStep objects +available for use in a Build, and the parameters which can be used to +control each. + +@menu +* Common Parameters:: +* Using Build Properties:: +* Source Checkout:: +* ShellCommand:: +* Simple ShellCommand Subclasses:: +* Python BuildSteps:: +* Transferring Files:: +* Steps That Run on the Master:: +* Triggering Schedulers:: +* Writing New BuildSteps:: +@end menu + +@node Common Parameters +@subsection Common Parameters + +The standard @code{Build} runs a series of @code{BuildStep}s in order, +only stopping when it runs out of steps or if one of them requests +that the build be halted. It collects status information from each one +to create an overall build status (of SUCCESS, WARNINGS, or FAILURE). + +All BuildSteps accept some common parameters. Some of these control +how their individual status affects the overall build. Others are used +to specify which @code{Locks} (see @pxref{Interlocks}) should be +acquired before allowing the step to run. + +Arguments common to all @code{BuildStep} subclasses: + + +@table @code +@item name +the name used to describe the step on the status display. It is also +used to give a name to any LogFiles created by this step. + +@item haltOnFailure +if True, a FAILURE of this build step will cause the build to halt +immediately. Steps with @code{alwaysRun=True} are still run. Generally +speaking, haltOnFailure implies flunkOnFailure (the default for most +BuildSteps). In some cases, particularly series of tests, it makes sense +to haltOnFailure if something fails early on but not flunkOnFailure. +This can be achieved with haltOnFailure=True, flunkOnFailure=False. + +@item flunkOnWarnings +when True, a WARNINGS or FAILURE of this build step will mark the +overall build as FAILURE. The remaining steps will still be executed. + +@item flunkOnFailure +when True, a FAILURE of this build step will mark the overall build as +a FAILURE. The remaining steps will still be executed. + +@item warnOnWarnings +when True, a WARNINGS or FAILURE of this build step will mark the +overall build as having WARNINGS. The remaining steps will still be +executed. + +@item warnOnFailure +when True, a FAILURE of this build step will mark the overall build as +having WARNINGS. The remaining steps will still be executed. + +@item alwaysRun +if True, this build step will always be run, even if a previous buildstep +with @code{haltOnFailure=True} has failed. + +@item doStepIf +A step can be configured to only run under certain conditions. To do this, set +the step's @code{doStepIf} to a boolean value, or to a function that returns a +boolean value. If the value or function result is false, then the step will +return SKIPPED without doing anything. Oherwise, the step will be executed +normally. If you set @code{doStepIf} to a function, that function should +accept one parameter, which will be the @code{Step} object itself. + +@item locks +a list of Locks (instances of @code{buildbot.locks.SlaveLock} or +@code{buildbot.locks.MasterLock}) that should be acquired before +starting this Step. The Locks will be released when the step is +complete. Note that this is a list of actual Lock instances, not +names. Also note that all Locks must have unique names. + +@end table + +@node Using Build Properties +@subsection Using Build Properties +@cindex Properties + +Build properties are a generalized way to provide configuration +information to build steps; see @ref{Build Properties}. + +Some build properties are inherited from external sources -- global +properties, schedulers, or buildslaves. Some build properties are +set when the build starts, such as the SourceStamp information. Other +properties can be set by BuildSteps as they run, for example the +various Source steps will set the @code{got_revision} property to the +source revision that was actually checked out (which can be useful +when the SourceStamp in use merely requested the ``latest revision'': +@code{got_revision} will tell you what was actually built). + +In custom BuildSteps, you can get and set the build properties with +the @code{getProperty}/@code{setProperty} methods. Each takes a string +for the name of the property, and returns or accepts an +arbitrary@footnote{Build properties are serialized along with the +build results, so they must be serializable. For this reason, the +value of any build property should be simple inert data: strings, +numbers, lists, tuples, and dictionaries. They should not contain +class instances.} object. For example: + +@example +class MakeTarball(ShellCommand): + def start(self): + if self.getProperty("os") == "win": + self.setCommand([ ... ]) # windows-only command + else: + self.setCommand([ ... ]) # equivalent for other systems + ShellCommand.start(self) +@end example + +@heading WithProperties +@cindex WithProperties + +You can use build properties in ShellCommands by using the +@code{WithProperties} wrapper when setting the arguments of +the ShellCommand. This interpolates the named build properties +into the generated shell command. Most step parameters accept +@code{WithProperties}. Please file bugs for any parameters which +do not. + +@example +from buildbot.steps.shell import ShellCommand +from buildbot.process.properties import WithProperties + +f.addStep(ShellCommand( + command=["tar", "czf", + WithProperties("build-%s.tar.gz", "revision"), + "source"])) +@end example + +If this BuildStep were used in a tree obtained from Subversion, it +would create a tarball with a name like @file{build-1234.tar.gz}. + +The @code{WithProperties} function does @code{printf}-style string +interpolation, using strings obtained by calling +@code{build.getProperty(propname)}. Note that for every @code{%s} (or +@code{%d}, etc), you must have exactly one additional argument to +indicate which build property you want to insert. + +You can also use python dictionary-style string interpolation by using +the @code{%(propname)s} syntax. In this form, the property name goes +in the parentheses, and WithProperties takes @emph{no} additional +arguments: + +@example +f.addStep(ShellCommand( + command=["tar", "czf", + WithProperties("build-%(revision)s.tar.gz"), + "source"])) +@end example + +Don't forget the extra ``s'' after the closing parenthesis! This is +the cause of many confusing errors. + +The dictionary-style interpolation supports a number of more advanced +syntaxes, too. + +@table @code + +@item propname:-replacement +If @code{propname} exists, substitute its value; otherwise, +substitute @code{replacement}. @code{replacement} may be empty +(@code{%(propname:-)s}) + +@item propname:+replacement +If @code{propname} exists, substitute @code{replacement}; otherwise, +substitute an empty string. + +@end table + +Although these are similar to shell substitutions, no other +substitutions are currently supported, and @code{replacement} in the +above cannot contain more substitutions. + +Note: like python, you can either do positional-argument interpolation +@emph{or} keyword-argument interpolation, not both. Thus you cannot use +a string like @code{WithProperties("foo-%(revision)s-%s", "branch")}. + +@heading Common Build Properties + +The following build properties are set when the build is started, and +are available to all steps. + +@table @code +@item branch + +This comes from the build's SourceStamp, and describes which branch is +being checked out. This will be @code{None} (which interpolates into +@code{WithProperties} as an empty string) if the build is on the +default branch, which is generally the trunk. Otherwise it will be a +string like ``branches/beta1.4''. The exact syntax depends upon the VC +system being used. + +@item revision + +This also comes from the SourceStamp, and is the revision of the source code +tree that was requested from the VC system. When a build is requested of a +specific revision (as is generally the case when the build is triggered by +Changes), this will contain the revision specification. This is always a +string, although the syntax depends upon the VC system in use: for SVN it is an +integer, for Mercurial it is a short string, for Darcs it is a rather large +string, etc. + +If the ``force build'' button was pressed, the revision will be @code{None}, +which means to use the most recent revision available. This is a ``trunk +build''. This will be interpolated as an empty string. + +@item got_revision + +This is set when a Source step checks out the source tree, and +provides the revision that was actually obtained from the VC system. +In general this should be the same as @code{revision}, except for +trunk builds, where @code{got_revision} indicates what revision was +current when the checkout was performed. This can be used to rebuild +the same source code later. + +Note that for some VC systems (Darcs in particular), the revision is a +large string containing newlines, and is not suitable for interpolation +into a filename. + +@item buildername + +This is a string that indicates which Builder the build was a part of. +The combination of buildername and buildnumber uniquely identify a +build. + +@item buildnumber + +Each build gets a number, scoped to the Builder (so the first build +performed on any given Builder will have a build number of 0). This +integer property contains the build's number. + +@item slavename + +This is a string which identifies which buildslave the build is +running on. + +@item scheduler + +If the build was started from a scheduler, then this property will +contain the name of that scheduler. + +@end table + + +@node Source Checkout +@subsection Source Checkout + +The first step of any build is typically to acquire the source code +from which the build will be performed. There are several classes to +handle this, one for each of the different source control system that +Buildbot knows about. For a description of how Buildbot treats source +control in general, see @ref{Version Control Systems}. + +All source checkout steps accept some common parameters to control how +they get the sources and where they should be placed. The remaining +per-VC-system parameters are mostly to specify where exactly the +sources are coming from. + +@table @code +@item mode + +a string describing the kind of VC operation that is desired. Defaults +to @code{update}. + +@table @code +@item update +specifies that the CVS checkout/update should be performed directly +into the workdir. Each build is performed in the same directory, +allowing for incremental builds. This minimizes disk space, bandwidth, +and CPU time. However, it may encounter problems if the build process +does not handle dependencies properly (sometimes you must do a ``clean +build'' to make sure everything gets compiled), or if source files are +deleted but generated files can influence test behavior (e.g. python's +.pyc files), or when source directories are deleted but generated +files prevent CVS from removing them. Builds ought to be correct +regardless of whether they are done ``from scratch'' or incrementally, +but it is useful to test both kinds: this mode exercises the +incremental-build style. + +@item copy +specifies that the CVS workspace should be maintained in a separate +directory (called the 'copydir'), using checkout or update as +necessary. For each build, a new workdir is created with a copy of the +source tree (rm -rf workdir; cp -r copydir workdir). This doubles the +disk space required, but keeps the bandwidth low (update instead of a +full checkout). A full 'clean' build is performed each time. This +avoids any generated-file build problems, but is still occasionally +vulnerable to CVS problems such as a repository being manually +rearranged, causing CVS errors on update which are not an issue with a +full checkout. + +@c TODO: something is screwy about this, revisit. Is it the source +@c directory or the working directory that is deleted each time? + +@item clobber +specifes that the working directory should be deleted each time, +necessitating a full checkout for each build. This insures a clean +build off a complete checkout, avoiding any of the problems described +above. This mode exercises the ``from-scratch'' build style. + +@item export +this is like @code{clobber}, except that the 'cvs export' command is +used to create the working directory. This command removes all CVS +metadata files (the CVS/ directories) from the tree, which is +sometimes useful for creating source tarballs (to avoid including the +metadata in the tar file). +@end table + +@item workdir +like all Steps, this indicates the directory where the build will take +place. Source Steps are special in that they perform some operations +outside of the workdir (like creating the workdir itself). + +@item alwaysUseLatest +if True, bypass the usual ``update to the last Change'' behavior, and +always update to the latest changes instead. + +@item retry +If set, this specifies a tuple of @code{(delay, repeats)} which means +that when a full VC checkout fails, it should be retried up to +@var{repeats} times, waiting @var{delay} seconds between attempts. If +you don't provide this, it defaults to @code{None}, which means VC +operations should not be retried. This is provided to make life easier +for buildslaves which are stuck behind poor network connections. + +@end table + + +My habit as a developer is to do a @code{cvs update} and @code{make} each +morning. Problems can occur, either because of bad code being checked in, or +by incomplete dependencies causing a partial rebuild to fail where a +complete from-scratch build might succeed. A quick Builder which emulates +this incremental-build behavior would use the @code{mode='update'} +setting. + +On the other hand, other kinds of dependency problems can cause a clean +build to fail where a partial build might succeed. This frequently results +from a link step that depends upon an object file that was removed from a +later version of the tree: in the partial tree, the object file is still +around (even though the Makefiles no longer know how to create it). + +``official'' builds (traceable builds performed from a known set of +source revisions) are always done as clean builds, to make sure it is +not influenced by any uncontrolled factors (like leftover files from a +previous build). A ``full'' Builder which behaves this way would want +to use the @code{mode='clobber'} setting. + +Each VC system has a corresponding source checkout class: their +arguments are described on the following pages. + + +@menu +* CVS:: +* SVN:: +* Darcs:: +* Mercurial:: +* Arch:: +* Bazaar:: +* Bzr:: +* P4:: +* Git:: +* BitKeeper:: +@end menu + +@node CVS +@subsubsection CVS +@cindex CVS Checkout +@bsindex buildbot.steps.source.CVS + + +The @code{CVS} build step performs a @uref{http://www.nongnu.org/cvs/, +CVS} checkout or update. It takes the following arguments: + +@table @code +@item cvsroot +(required): specify the CVSROOT value, which points to a CVS +repository, probably on a remote machine. For example, the cvsroot +value you would use to get a copy of the Buildbot source code is +@code{:pserver:anonymous@@cvs.sourceforge.net:/cvsroot/buildbot} + +@item cvsmodule +(required): specify the cvs @code{module}, which is generally a +subdirectory of the CVSROOT. The cvsmodule for the Buildbot source +code is @code{buildbot}. + +@item branch +a string which will be used in a @code{-r} argument. This is most +useful for specifying a branch to work on. Defaults to @code{HEAD}. + +@item global_options +a list of flags to be put before the verb in the CVS command. + +@item checkoutDelay +if set, the number of seconds to put between the timestamp of the last +known Change and the value used for the @code{-D} option. Defaults to +half of the parent Build's treeStableTimer. + +@end table + + +@node SVN +@subsubsection SVN + +@cindex SVN Checkout +@bsindex buildbot.steps.source.SVN + + +The @code{SVN} build step performs a +@uref{http://subversion.tigris.org, Subversion} checkout or update. +There are two basic ways of setting up the checkout step, depending +upon whether you are using multiple branches or not. + +If all of your builds use the same branch, then you should create the +@code{SVN} step with the @code{svnurl} argument: + +@table @code +@item svnurl +(required): this specifies the @code{URL} argument that will be given +to the @code{svn checkout} command. It dictates both where the +repository is located and which sub-tree should be extracted. In this +respect, it is like a combination of the CVS @code{cvsroot} and +@code{cvsmodule} arguments. For example, if you are using a remote +Subversion repository which is accessible through HTTP at a URL of +@code{http://svn.example.com/repos}, and you wanted to check out the +@code{trunk/calc} sub-tree, you would use +@code{svnurl="http://svn.example.com/repos/trunk/calc"} as an argument +to your @code{SVN} step. +@end table + +If, on the other hand, you are building from multiple branches, then +you should create the @code{SVN} step with the @code{baseURL} and +@code{defaultBranch} arguments instead: + +@table @code +@item baseURL +(required): this specifies the base repository URL, to which a branch +name will be appended. It should probably end in a slash. + +@item defaultBranch +(optional): this specifies the name of the branch to use when a Build +does not provide one of its own. This will be appended to +@code{baseURL} to create the string that will be passed to the +@code{svn checkout} command. + +@item username +(optional): if specified, this will be passed to the @code{svn} +binary with a @code{--username} option. + +@item password +(optional): if specified, this will be passed to the @code{svn} +binary with a @code{--password} option. The password itself will be +suitably obfuscated in the logs. + +@item extra_args +(optional): if specified, an array of strings that will be passed as +extra arguments to the @code{svn} binary. + +@item keep_on_purge +(optional): specific files or directories to keep between purges, +like some build outputs that can be reused between builds. + +@item ignore_ignores +(optional): when purging changes, don't use rules defined in +svn:ignore properties and global-ignores in subversion/config. + +@item always_purge +(optional): if set to True, always purge local changes after each +build. This is everything that would appear in a @code{svn status}. + +@item depth +(optional): Specify depth argument to achieve sparse checkout. Only available if slave has Subversion 1.5 or higher. + +If set to "empty" updates will not pull in any files or subdirectories not already present. If set to "files", updates will pull in any files not already present, but not directories. If set to "immediates", updates willl pull in any files or subdirectories not already present, the new subdirectories will have depth: empty. If set to "infinity", updates will pull in any files or subdirectories not already present; the new subdirectories will have depth-infinity. Infinity is equivalent to SVN default update behavior, without specifying any depth argument. + +@end table + +If you are using branches, you must also make sure your +@code{ChangeSource} will report the correct branch names. + +@heading branch example + +Let's suppose that the ``MyProject'' repository uses branches for the +trunk, for various users' individual development efforts, and for +several new features that will require some amount of work (involving +multiple developers) before they are ready to merge onto the trunk. +Such a repository might be organized as follows: + +@example +svn://svn.example.org/MyProject/trunk +svn://svn.example.org/MyProject/branches/User1/foo +svn://svn.example.org/MyProject/branches/User1/bar +svn://svn.example.org/MyProject/branches/User2/baz +svn://svn.example.org/MyProject/features/newthing +svn://svn.example.org/MyProject/features/otherthing +@end example + +Further assume that we want the Buildbot to run tests against the +trunk and against all the feature branches (i.e., do a +checkout/compile/build of branch X when a file has been changed on +branch X, when X is in the set [trunk, features/newthing, +features/otherthing]). We do not want the Buildbot to automatically +build any of the user branches, but it should be willing to build a +user branch when explicitly requested (most likely by the user who +owns that branch). + +There are three things that need to be set up to accomodate this +system. The first is a ChangeSource that is capable of identifying the +branch which owns any given file. This depends upon a user-supplied +function, in an external program that runs in the SVN commit hook and +connects to the buildmaster's @code{PBChangeSource} over a TCP +connection. (you can use the ``@code{buildbot sendchange}'' utility +for this purpose, but you will still need an external program to +decide what value should be passed to the @code{--branch=} argument). +For example, a change to a file with the SVN url of +``svn://svn.example.org/MyProject/features/newthing/src/foo.c'' should +be broken down into a Change instance with +@code{branch='features/newthing'} and @code{file='src/foo.c'}. + +The second piece is an @code{AnyBranchScheduler} which will pay +attention to the desired branches. It will not pay attention to the +user branches, so it will not automatically start builds in response +to changes there. The AnyBranchScheduler class requires you to +explicitly list all the branches you want it to use, but it would not +be difficult to write a subclass which used +@code{branch.startswith('features/'} to remove the need for this +explicit list. Or, if you want to build user branches too, you can use +AnyBranchScheduler with @code{branches=None} to indicate that you want +it to pay attention to all branches. + +The third piece is an @code{SVN} checkout step that is configured to +handle the branches correctly, with a @code{baseURL} value that +matches the way the ChangeSource splits each file's URL into base, +branch, and file. + +@example +from buildbot.changes.pb import PBChangeSource +from buildbot.scheduler import AnyBranchScheduler +from buildbot.process import source, factory +from buildbot.steps import source, shell + +c['change_source'] = PBChangeSource() +s1 = AnyBranchScheduler('main', + ['trunk', 'features/newthing', 'features/otherthing'], + 10*60, ['test-i386', 'test-ppc']) +c['schedulers'] = [s1] + +f = factory.BuildFactory() +f.addStep(source.SVN(mode='update', + baseURL='svn://svn.example.org/MyProject/', + defaultBranch='trunk')) +f.addStep(shell.Compile(command="make all")) +f.addStep(shell.Test(command="make test")) + +c['builders'] = [ + @{'name':'test-i386', 'slavename':'bot-i386', 'builddir':'test-i386', + 'factory':f @}, + @{'name':'test-ppc', 'slavename':'bot-ppc', 'builddir':'test-ppc', + 'factory':f @}, + ] +@end example + +In this example, when a change arrives with a @code{branch} attribute +of ``trunk'', the resulting build will have an SVN step that +concatenates ``svn://svn.example.org/MyProject/'' (the baseURL) with +``trunk'' (the branch name) to get the correct svn command. If the +``newthing'' branch has a change to ``src/foo.c'', then the SVN step +will concatenate ``svn://svn.example.org/MyProject/'' with +``features/newthing'' to get the svnurl for checkout. + +@node Darcs +@subsubsection Darcs + +@cindex Darcs Checkout +@bsindex buildbot.steps.source.Darcs + + +The @code{Darcs} build step performs a +@uref{http://darcs.net/, Darcs} checkout or update. + +Like @xref{SVN}, this step can either be configured to always check +out a specific tree, or set up to pull from a particular branch that +gets specified separately for each build. Also like SVN, the +repository URL given to Darcs is created by concatenating a +@code{baseURL} with the branch name, and if no particular branch is +requested, it uses a @code{defaultBranch}. The only difference in +usage is that each potential Darcs repository URL must point to a +fully-fledged repository, whereas SVN URLs usually point to sub-trees +of the main Subversion repository. In other words, doing an SVN +checkout of @code{baseURL} is legal, but silly, since you'd probably +wind up with a copy of every single branch in the whole repository. +Doing a Darcs checkout of @code{baseURL} is just plain wrong, since +the parent directory of a collection of Darcs repositories is not +itself a valid repository. + +The Darcs step takes the following arguments: + +@table @code +@item repourl +(required unless @code{baseURL} is provided): the URL at which the +Darcs source repository is available. + +@item baseURL +(required unless @code{repourl} is provided): the base repository URL, +to which a branch name will be appended. It should probably end in a +slash. + +@item defaultBranch +(allowed if and only if @code{baseURL} is provided): this specifies +the name of the branch to use when a Build does not provide one of its +own. This will be appended to @code{baseURL} to create the string that +will be passed to the @code{darcs get} command. +@end table + +@node Mercurial +@subsubsection Mercurial + +@cindex Mercurial Checkout +@bsindex buildbot.steps.source.Mercurial + + +The @code{Mercurial} build step performs a +@uref{http://selenic.com/mercurial, Mercurial} (aka ``hg'') checkout +or update. + +Branches are available in two modes: ''dirname'' like @xref{Darcs}, or +''inrepo'', which uses the repository internal branches. Make sure this +setting matches your changehook, if you have that installed. + +The Mercurial step takes the following arguments: + +@table @code +@item repourl +(required unless @code{baseURL} is provided): the URL at which the +Mercurial source repository is available. + +@item baseURL +(required unless @code{repourl} is provided): the base repository URL, +to which a branch name will be appended. It should probably end in a +slash. + +@item defaultBranch +(allowed if and only if @code{baseURL} is provided): this specifies +the name of the branch to use when a Build does not provide one of its +own. This will be appended to @code{baseURL} to create the string that +will be passed to the @code{hg clone} command. + +@item branchType +either 'dirname' (default) or 'inrepo' depending on whether +the branch name should be appended to the @code{baseURL} +or the branch is a mercurial named branch and can be +found within the @code{repourl}. + +@item clobberOnBranchChange +boolean, defaults to True. If set and +using inrepos branches, clobber the tree +at each branch change. Otherwise, just +update to the branch. + +@end table + + +@node Arch +@subsubsection Arch + +@cindex Arch Checkout +@bsindex buildbot.steps.source.Arch + + +The @code{Arch} build step performs an @uref{http://gnuarch.org/, +Arch} checkout or update using the @code{tla} client. It takes the +following arguments: + +@table @code +@item url +(required): this specifies the URL at which the Arch source archive is +available. + +@item version +(required): this specifies which ``development line'' (like a branch) +should be used. This provides the default branch name, but individual +builds may specify a different one. + +@item archive +(optional): Each repository knows its own archive name. If this +parameter is provided, it must match the repository's archive name. +The parameter is accepted for compatibility with the @code{Bazaar} +step, below. + +@end table + +@node Bazaar +@subsubsection Bazaar + +@cindex Bazaar Checkout +@bsindex buildbot.steps.source.Bazaar + + +@code{Bazaar} is an alternate implementation of the Arch VC system, +which uses a client named @code{baz}. The checkout semantics are just +different enough from @code{tla} that there is a separate BuildStep for +it. + +It takes exactly the same arguments as @code{Arch}, except that the +@code{archive=} parameter is required. (baz does not emit the archive +name when you do @code{baz register-archive}, so we must provide it +ourselves). + + +@node Bzr +@subsubsection Bzr + +@cindex Bzr Checkout +@bsindex buildbot.steps.source.Bzr + +@code{bzr} is a descendant of Arch/Baz, and is frequently referred to +as simply ``Bazaar''. The repository-vs-workspace model is similar to +Darcs, but it uses a strictly linear sequence of revisions (one +history per branch) like Arch. Branches are put in subdirectories. +This makes it look very much like Mercurial. It takes the following +arguments: + +@table @code + +@item repourl +(required unless @code{baseURL} is provided): the URL at which the +Bzr source repository is available. + +@item baseURL +(required unless @code{repourl} is provided): the base repository URL, +to which a branch name will be appended. It should probably end in a +slash. + +@item defaultBranch +(allowed if and only if @code{baseURL} is provided): this specifies +the name of the branch to use when a Build does not provide one of its +own. This will be appended to @code{baseURL} to create the string that +will be passed to the @code{bzr checkout} command. + +@item forceSharedRepo +(boolean, optional, defaults to False): If set to True, the working directory +will be made into a bzr shared repository if it is not already. Shared +repository greatly reduces the amount of history data that needs to be +downloaded if not using update/copy mode, or if using update/copy mode with +multiple branches. +@end table + + + +@node P4 +@subsubsection P4 + +@cindex Perforce Update +@bsindex buildbot.steps.source.P4 +@c TODO @bsindex buildbot.steps.source.P4Sync + + +The @code{P4} build step creates a @uref{http://www.perforce.com/, +Perforce} client specification and performs an update. + +@table @code +@item p4base +A view into the Perforce depot without branch name or trailing "...". +Typically "//depot/proj/". +@item defaultBranch +A branch name to append on build requests if none is specified. +Typically "trunk". +@item p4port +(optional): the host:port string describing how to get to the P4 Depot +(repository), used as the -p argument for all p4 commands. +@item p4user +(optional): the Perforce user, used as the -u argument to all p4 +commands. +@item p4passwd +(optional): the Perforce password, used as the -p argument to all p4 +commands. +@item p4extra_views +(optional): a list of (depotpath, clientpath) tuples containing extra +views to be mapped into the client specification. Both will have +"/..." appended automatically. The client name and source directory +will be prepended to the client path. +@item p4client +(optional): The name of the client to use. In mode='copy' and +mode='update', it's particularly important that a unique name is used +for each checkout directory to avoid incorrect synchronization. For +this reason, Python percent substitution will be performed on this value +to replace %(slave)s with the slave name and %(builder)s with the +builder name. The default is "buildbot_%(slave)s_%(build)s". +@end table + + +@node Git +@subsubsection Git + +@cindex Git Checkout +@bsindex buildbot.steps.source.Git + +The @code{Git} build step clones or updates a @uref{http://git.or.cz/, +Git} repository and checks out the specified branch or revision. Note +that the buildbot supports Git version 1.2.0 and later: earlier +versions (such as the one shipped in Ubuntu 'Dapper') do not support +the @command{git init} command that the buildbot uses. + +The Git step takes the following arguments: + +@table @code +@item repourl +(required): the URL of the upstream Git repository. + +@item branch +(optional): this specifies the name of the branch to use when a Build +does not provide one of its own. If this this parameter is not +specified, and the Build does not provide a branch, the ``master'' +branch will be used. + +@item ignore_ignores +(optional): when purging changes, don't use .gitignore and +.git/info/exclude. + +@end table + + +@node BitKeeper +@subsubsection BitKeeper + +@cindex BitKeeper Checkout +@bsindex buildbot.steps.source.BK + +The @code{BK} build step performs a @uref{http://www.bitkeeper.com/, BitKeeper} +checkout or update. + +The BitKeeper step takes the following arguments: + +@table @code +@item repourl +(required unless @code{baseURL} is provided): the URL at which the +BitKeeper source repository is available. + +@item baseURL +(required unless @code{repourl} is provided): the base repository URL, +to which a branch name will be appended. It should probably end in a +slash. + +@end table + + +@node ShellCommand +@subsection ShellCommand + +@bsindex buildbot.steps.shell.ShellCommand +@c TODO @bsindex buildbot.steps.shell.TreeSize + +This is a useful base class for just about everything you might want +to do during a build (except for the initial source checkout). It runs +a single command in a child shell on the buildslave. All stdout/stderr +is recorded into a LogFile. The step finishes with a status of FAILURE +if the command's exit code is non-zero, otherwise it has a status of +SUCCESS. + +The preferred way to specify the command is with a list of argv strings, +since this allows for spaces in filenames and avoids doing any fragile +shell-escaping. You can also specify the command with a single string, in +which case the string is given to '/bin/sh -c COMMAND' for parsing. + +On Windows, commands are run via @code{cmd.exe /c} which works well. However, +if you're running a batch file, the error level does not get propagated +correctly unless you add 'call' before your batch file's name: +@code{cmd=['call', 'myfile.bat', ...]}. + +@code{ShellCommand} arguments: + +@table @code +@item command +a list of strings (preferred) or single string (discouraged) which +specifies the command to be run. A list of strings is preferred +because it can be used directly as an argv array. Using a single +string (with embedded spaces) requires the buildslave to pass the +string to /bin/sh for interpretation, which raises all sorts of +difficult questions about how to escape or interpret shell +metacharacters. + +@item workdir +All ShellCommands are run by default in the ``workdir'', which +defaults to the ``@file{build}'' subdirectory of the slave builder's +base directory. The absolute path of the workdir will thus be the +slave's basedir (set as an option to @code{buildbot create-slave}, +@pxref{Creating a buildslave}) plus the builder's basedir (set in the +builder's @code{c['builddir']} key in master.cfg) plus the workdir +itself (a class-level attribute of the BuildFactory, defaults to +``@file{build}''). + +For example: + +@example +f.addStep(ShellCommand(command=["make", "test"], + workdir="build/tests")) +@end example + +@item env +a dictionary of environment strings which will be added to the child +command's environment. For example, to run tests with a different i18n +language setting, you might use + +@example +f.addStep(ShellCommand(command=["make", "test"], + env=@{'LANG': 'fr_FR'@})) +@end example + +These variable settings will override any existing ones in the +buildslave's environment or the environment specified in the +Builder. The exception is PYTHONPATH, which is merged +with (actually prepended to) any existing $PYTHONPATH setting. The +value is treated as a list of directories to prepend, and a single +string is treated like a one-item list. For example, to prepend both +@file{/usr/local/lib/python2.3} and @file{/home/buildbot/lib/python} +to any existing $PYTHONPATH setting, you would do something like the +following: + +@example +f.addStep(ShellCommand( + command=["make", "test"], + env=@{'PYTHONPATH': ["/usr/local/lib/python2.3", + "/home/buildbot/lib/python"] @})) +@end example + +@item want_stdout +if False, stdout from the child process is discarded rather than being +sent to the buildmaster for inclusion in the step's LogFile. + +@item want_stderr +like @code{want_stdout} but for stderr. Note that commands run through +a PTY do not have separate stdout/stderr streams: both are merged into +stdout. + +@item usePTY +Should this command be run in a @code{pty}? The default is to observe the +configuration of the client (@pxref{Buildslave Options}), but specifying +@code{True} or @code{False} here will override the default. + +The advantage of using a PTY is that ``grandchild'' processes are more likely +to be cleaned up if the build is interrupted or times out (since it enables the +use of a ``process group'' in which all child processes will be placed). The +disadvantages: some forms of Unix have problems with PTYs, some of your unit +tests may behave differently when run under a PTY (generally those which check +to see if they are being run interactively), and PTYs will merge the stdout and +stderr streams into a single output stream (which means the red-vs-black +coloring in the logfiles will be lost). + +@item logfiles +Sometimes commands will log interesting data to a local file, rather +than emitting everything to stdout or stderr. For example, Twisted's +``trial'' command (which runs unit tests) only presents summary +information to stdout, and puts the rest into a file named +@file{_trial_temp/test.log}. It is often useful to watch these files +as the command runs, rather than using @command{/bin/cat} to dump +their contents afterwards. + +The @code{logfiles=} argument allows you to collect data from these +secondary logfiles in near-real-time, as the step is running. It +accepts a dictionary which maps from a local Log name (which is how +the log data is presented in the build results) to either a remote filename +(interpreted relative to the build's working directory), or a dictionary +of options. Each named file will be polled on a regular basis (every couple +of seconds) as the build runs, and any new text will be sent over to the +buildmaster. + +If you provide a dictionary of options instead of a string, you must specify +the @code{filename} key. You can optionally provide a @code{follow} key which +is a boolean controlling whether a logfile is followed or concatenated in its +entirety. Following is appropriate for logfiles to which the build step will +append, where the pre-existing contents are not interesting. The default value +for @code{follow} is @code{False}, which gives the same behavior as just +providing a string filename. + +@example +f.addStep(ShellCommand( + command=["make", "test"], + logfiles=@{"triallog": "_trial_temp/test.log"@})) +@end example + +@example +f.addStep(ShellCommand( + command=["make", "test"], + logfiles=@{"triallog": @{"filename": "_trial_temp/test.log", + "follow": True,@}@})) +@end example + + +@item lazylogfiles +If set to @code{True}, logfiles will be tracked lazily, meaning that they will +only be added when and if something is written to them. This can be used to +suppress the display of empty or missing log files. The default is @code{False}. + + +@item timeout +if the command fails to produce any output for this many seconds, it +is assumed to be locked up and will be killed. + +@item maxTime +if the command takes longer than this many seconds, it will be killed. + +@item description +This will be used to describe the command (on the Waterfall display) +while the command is still running. It should be a single +imperfect-tense verb, like ``compiling'' or ``testing''. The preferred +form is a list of short strings, which allows the HTML +displays to create narrower columns by emitting a
tag between each +word. You may also provide a single string. + +@item descriptionDone +This will be used to describe the command once it has finished. A +simple noun like ``compile'' or ``tests'' should be used. Like +@code{description}, this may either be a list of short strings or a +single string. + +If neither @code{description} nor @code{descriptionDone} are set, the +actual command arguments will be used to construct the description. +This may be a bit too wide to fit comfortably on the Waterfall +display. + +@example +f.addStep(ShellCommand(command=["make", "test"], + description=["testing"], + descriptionDone=["tests"])) +@end example + +@item logEnviron +If this option is true (the default), then the step's logfile will describe the +environment variables on the slave. In situations where the environment is not +relevant and is long, it may be easier to set @code{logEnviron=False}. + +@end table + +@node Simple ShellCommand Subclasses +@subsection Simple ShellCommand Subclasses + +Several subclasses of ShellCommand are provided as starting points for +common build steps. These are all very simple: they just override a few +parameters so you don't have to specify them yourself, making the master.cfg +file less verbose. + +@menu +* Configure:: +* Compile:: +* Test:: +* TreeSize:: +* PerlModuleTest:: +* Testing with mysql-test-run:: +* SetProperty:: +* SubunitShellCommand:: +@end menu + +@node Configure +@subsubsection Configure + +@bsindex buildbot.steps.shell.Configure + +This is intended to handle the @code{./configure} step from +autoconf-style projects, or the @code{perl Makefile.PL} step from perl +MakeMaker.pm-style modules. The default command is @code{./configure} +but you can change this by providing a @code{command=} parameter. + +@node Compile +@subsubsection Compile + +@bsindex buildbot.steps.shell.Compile + +This is meant to handle compiling or building a project written in C. +The default command is @code{make all}. When the compile is finished, +the log file is scanned for GCC warning messages, a summary log is +created with any problems that were seen, and the step is marked as +WARNINGS if any were discovered. Through the @code{WarningCountingShellCommand} +superclass, the number of warnings is stored in a Build Property named +``warnings-count'', which is accumulated over all Compile steps (so if two +warnings are found in one step, and three are found in another step, the +overall build will have a ``warnings-count'' property of 5). + +The default regular expression used to detect a warning is +@code{'.*warning[: ].*'} , which is fairly liberal and may cause +false-positives. To use a different regexp, provide a +@code{warningPattern=} argument, or use a subclass which sets the +@code{warningPattern} attribute: + +@example +f.addStep(Compile(command=["make", "test"], + warningPattern="^Warning: ")) +@end example + +The @code{warningPattern=} can also be a pre-compiled python regexp +object: this makes it possible to add flags like @code{re.I} (to use +case-insensitive matching). + +Note that the compiled @code{warningPattern} will have its @code{match} method +called, which is subtly different from a @code{search}. Your regular +expression must match the from the beginning of the line. This means that to +look for the word "warning" in the middle of a line, you will need to +prepend @code{'.*'} to your regular expression. + +The @code{suppressionFile=} argument can be specified as the (relative) path +of a file inside the workdir defining warnings to be suppressed from the +warning counting and log file. The file will be uploaded to the master from +the slave before compiling, and any warning matched by a line in the +suppression file will be ignored. This is useful to accept certain warnings +(eg. in some special module of the source tree or in cases where the compiler +is being particularly stupid), yet still be able to easily detect and fix the +introduction of new warnings. + +The file must contain one line per pattern of warnings to ignore. Empty lines +and lines beginning with @code{#} are ignored. Other lines must consist of a +regexp matching the file name, followed by a colon (@code{:}), followed by a +regexp matching the text of the warning. Optionally this may be followed by +another colon and a line number range. For example: + +@example +# Sample warning suppression file + +mi_packrec.c : .*result of 32-bit shift implicitly converted to 64 bits.* : 560-600 +DictTabInfo.cpp : .*invalid access to non-static.* +kernel_types.h : .*only defines private constructors and has no friends.* : 51 +@end example + +If no line number range is specified, the pattern matches the whole file; if +only one number is given it matches only on that line. + +The default warningPattern regexp only matches the warning text, so line +numbers and file names are ignored. To enable line number and file name +matching, privide a different regexp and provide a function (callable) as the +argument of @code{warningExtractor=}. The function is called with three +arguments: the BuildStep object, the line in the log file with the warning, +and the @code{SRE_Match} object of the regexp search for @code{warningPattern}. It +should return a tuple @code{(filename, linenumber, warning_test)}. For +example: + +@example +f.addStep(Compile(command=["make"], + warningPattern="^(.*?):([0-9]+): [Ww]arning: (.*)$", + warningExtractor=Compile.warnExtractFromRegexpGroups, + suppressionFile="support-files/compiler_warnings.supp")) +@end example + +(@code{Compile.warnExtractFromRegexpGroups} is a pre-defined function that +returns the filename, linenumber, and text from groups (1,2,3) of the regexp +match). + +In projects with source files in multiple directories, it is possible to get +full path names for file names matched in the suppression file, as long as the +build command outputs the names of directories as they are entered into and +left again. For this, specify regexps for the arguments +@code{directoryEnterPattern=} and @code{directoryLeavePattern=}. The +@code{directoryEnterPattern=} regexp should return the name of the directory +entered into in the first matched group. The defaults, which are suitable for +GNU Make, are these: + +@example +directoryEnterPattern = "make.*: Entering directory [\"`'](.*)['`\"]" +directoryLeavePattern = "make.*: Leaving directory" +@end example + +(TODO: this step needs to be extended to look for GCC error messages +as well, and collect them into a separate logfile, along with the +source code filenames involved). + + +@node Test +@subsubsection Test + +@bsindex buildbot.steps.shell.Test + +This is meant to handle unit tests. The default command is @code{make +test}, and the @code{warnOnFailure} flag is set. + +@node TreeSize +@subsubsection TreeSize + +@bsindex buildbot.steps.shell.TreeSize + +This is a simple command that uses the 'du' tool to measure the size +of the code tree. It puts the size (as a count of 1024-byte blocks, +aka 'KiB' or 'kibibytes') on the step's status text, and sets a build +property named 'tree-size-KiB' with the same value. + +@node PerlModuleTest +@subsubsection PerlModuleTest + +@bsindex buildbot.steps.shell.PerlModuleTest + +This is a simple command that knows how to run tests of perl modules. +It parses the output to determine the number of tests passed and +failed and total number executed, saving the results for later query. + +@node Testing with mysql-test-run +@subsubsection Testing with mysql-test-run + +The @code{process.mtrlogobserver.MTR} class is a subclass of @code{Test} +(@ref{Test}). It is used to run test suites using the mysql-test-run program, +as used in MySQL, Drizzle, MariaDB, and MySQL storage engine plugins. + +The shell command to run the test suite is specified in the same way as for +the Test class. The MTR class will parse the output of running the test suite, +and use the count of tests executed so far to provide more accurate completion +time estimates. Any test failures that occur during the test are summarized on +the Waterfall Display. + +Server error logs are added as additional log files, useful to debug test +failures. + +Optionally, data about the test run and any test failures can be inserted into +a database for further analysis and report generation. To use this facility, +create an instance of @code{twisted.enterprise.adbapi.ConnectionPool} with +connections to the database. The necessary tables can be created automatically +by setting @code{autoCreateTables} to @code{True}, or manually using the SQL +found in the @file{mtrlogobserver.py} source file. + +One problem with specifying a database is that each reload of the +configuration will get a new instance of @code{ConnectionPool} (even if the +connection parameters are the same). To avoid that Buildbot thinks the builder +configuration has changed because of this, use the +@code{process.mtrlogobserver.EqConnectionPool} subclass of +@code{ConnectionPool}, which implements an equiality operation that avoids +this problem. + +Example use: + +@example +from buildbot.process.mtrlogobserver import MTR, EqConnectionPool +myPool = EqConnectionPool("MySQLdb", "host", "buildbot", "password", "db") +myFactory.addStep(MTR(workdir="mysql-test", dbpool=myPool, + command=["perl", "mysql-test-run.pl", "--force"])) +@end example + +@code{MTR} arguments: + +@table @code + +@item textLimit +Maximum number of test failures to show on the waterfall page (to not flood +the page in case of a large number of test failures. Defaults to 5. + +@item testNameLimit +Maximum length of test names to show unabbreviated in the waterfall page, to +avoid excessive column width. Defaults to 16. + +@item parallel +Value of @code{--parallel} option used for mysql-test-run.pl (number of processes +used to run the test suite in parallel). Defaults to 4. This is used to +determine the number of server error log files to download from the +slave. Specifying a too high value does not hurt (as nonexisting error logs +will be ignored), however if using @code{--parallel} value greater than the default +it needs to be specified, or some server error logs will be missing. + +@item dbpool +An instance of twisted.enterprise.adbapi.ConnectionPool, or None. Defaults to +None. If specified, results are inserted into the database using the +ConnectionPool. + +@item autoCreateTables +Boolean, defaults to False. If True (and @code{dbpool} is specified), the +necessary database tables will be created automatically if they do not exist +already. Alternatively, the tables can be created manually from the SQL +statements found in the mtrlogobserver.py source file. + +@item test_type +Short string that will be inserted into the database in the row for the test +run. Defaults to the empty string, but can be specified to identify different +types of test runs. + +@item test_info +Descriptive string that will be inserted into the database in the row for the test +run. Defaults to the empty string, but can be specified as a user-readable +description of this particular test run. + +@item mtr_subdir +The subdirectory in which to look for server error log files. Defaults to +``mysql-test'', which is usually correct. WithProperties is supported. + +@end table + + +@node SetProperty +@subsubsection SetProperty + +@bsindex buildbot.steps.shell.SetProperty + +This buildstep is similar to ShellCommand, except that it captures the +output of the command into a property. It is usually used like this: + +@example +f.addStep(SetProperty(command="uname -a", property="uname")) +@end example + +This runs @code{uname -a} and captures its stdout, stripped of leading +and trailing whitespace, in the property "uname". To avoid stripping, +add @code{strip=False}. The @code{property} argument can be specified +as a @code{WithProperties} object. + +The more advanced usage allows you to specify a function to extract +properties from the command output. Here you can use regular +expressions, string interpolation, or whatever you would like. +The function is called with three arguments: the exit status of the +command, its standard output as a string, and its standard error as +a string. It should return a dictionary containing all new properties. + +@example +def glob2list(rc, stdout, stderr): + jpgs = [ l.strip() for l in stdout.split('\n') ] + return @{ 'jpgs' : jpgs @} +f.addStep(SetProperty(command="ls -1 *.jpg", extract_fn=glob2list)) +@end example + +Note that any ordering relationship of the contents of stdout and +stderr is lost. For example, given + +@example +f.addStep(SetProperty( + command="echo output1; echo error >&2; echo output2", + extract_fn=my_extract)) +@end example + +Then @code{my_extract} will see @code{stdout="output1\noutput2\n"} +and @code{stderr="error\n"}. + +@node SubunitShellCommand +@subsubsection SubunitShellCommand + +@bsindex buildbot.process.subunitlogger.SubunitShellCommand + +This buildstep is similar to ShellCommand, except that it runs the log content +through a subunit filter to extract test and failure counts. + +@example +f.addStep(SubunitShellCommand(command="make test")) +@end example + +This runs @code{make test} and filters it through subunit. The 'tests' and +'test failed' progress metrics will now accumulate test data from the test run. + +@node Python BuildSteps +@subsection Python BuildSteps + +Here are some BuildSteps that are specifcally useful for projects +implemented in Python. + +@menu +* BuildEPYDoc:: +* PyFlakes:: +* PyLint:: +@end menu + +@node BuildEPYDoc +@subsubsection BuildEPYDoc + +@bsindex buildbot.steps.python.BuildEPYDoc + +@url{http://epydoc.sourceforge.net/, epydoc} is a tool for generating +API documentation for Python modules from their docstrings. It reads +all the .py files from your source tree, processes the docstrings +therein, and creates a large tree of .html files (or a single .pdf +file). + +The @code{buildbot.steps.python.BuildEPYDoc} step will run +@command{epydoc} to produce this API documentation, and will count the +errors and warnings from its output. + +You must supply the command line to be used. The default is +@command{make epydocs}, which assumes that your project has a Makefile +with an ``epydocs'' target. You might wish to use something like +@command{epydoc -o apiref source/PKGNAME} instead. You might also want +to add @command{--pdf} to generate a PDF file instead of a large tree +of HTML files. + +The API docs are generated in-place in the build tree (under the +workdir, in the subdirectory controlled by the ``-o'' argument). To +make them useful, you will probably have to copy them to somewhere +they can be read. A command like @command{rsync -ad apiref/ +dev.example.com:~public_html/current-apiref/} might be useful. You +might instead want to bundle them into a tarball and publish it in the +same place where the generated install tarball is placed. + +@example +from buildbot.steps.python import BuildEPYDoc + +... +f.addStep(BuildEPYDoc(command=["epydoc", "-o", "apiref", "source/mypkg"])) +@end example + + +@node PyFlakes +@subsubsection PyFlakes + +@bsindex buildbot.steps.python.PyFlakes + +@url{http://divmod.org/trac/wiki/DivmodPyflakes, PyFlakes} is a tool +to perform basic static analysis of Python code to look for simple +errors, like missing imports and references of undefined names. It is +like a fast and simple form of the C ``lint'' program. Other tools +(like pychecker) provide more detailed results but take longer to run. + +The @code{buildbot.steps.python.PyFlakes} step will run pyflakes and +count the various kinds of errors and warnings it detects. + +You must supply the command line to be used. The default is +@command{make pyflakes}, which assumes you have a top-level Makefile +with a ``pyflakes'' target. You might want to use something like +@command{pyflakes .} or @command{pyflakes src}. + +@example +from buildbot.steps.python import PyFlakes + +... +f.addStep(PyFlakes(command=["pyflakes", "src"])) +@end example + +@node PyLint +@subsubsection PyLint + +@bsindex buildbot.steps.python.PyLint + +Similarly, the @code{buildbot.steps.python.PyLint} step will run pylint and +analyze the results. + +You must supply the command line to be used. There is no default. + +@example +from buildbot.steps.python import PyLint + +... +f.addStep(PyLint(command=["pylint", "src"])) +@end example + + +@node Transferring Files +@subsection Transferring Files + +@cindex File Transfer +@bsindex buildbot.steps.transfer.FileUpload +@bsindex buildbot.steps.transfer.FileDownload +@bsindex buildbot.steps.transfer.DirectoryUpload + +Most of the work involved in a build will take place on the +buildslave. But occasionally it is useful to do some work on the +buildmaster side. The most basic way to involve the buildmaster is +simply to move a file from the slave to the master, or vice versa. +There are a pair of BuildSteps named @code{FileUpload} and +@code{FileDownload} to provide this functionality. @code{FileUpload} +moves a file @emph{up to} the master, while @code{FileDownload} moves +a file @emph{down from} the master. + +As an example, let's assume that there is a step which produces an +HTML file within the source tree that contains some sort of generated +project documentation. We want to move this file to the buildmaster, +into a @file{~/public_html} directory, so it can be visible to +developers. This file will wind up in the slave-side working directory +under the name @file{docs/reference.html}. We want to put it into the +master-side @file{~/public_html/ref.html}. + +@example +from buildbot.steps.shell import ShellCommand +from buildbot.steps.transfer import FileUpload + +f.addStep(ShellCommand(command=["make", "docs"])) +f.addStep(FileUpload(slavesrc="docs/reference.html", + masterdest="~/public_html/ref.html")) +@end example + +The @code{masterdest=} argument will be passed to os.path.expanduser, +so things like ``~'' will be expanded properly. Non-absolute paths +will be interpreted relative to the buildmaster's base directory. +Likewise, the @code{slavesrc=} argument will be expanded and +interpreted relative to the builder's working directory. + + +To move a file from the master to the slave, use the +@code{FileDownload} command. For example, let's assume that some step +requires a configuration file that, for whatever reason, could not be +recorded in the source code repository or generated on the buildslave +side: + +@example +from buildbot.steps.shell import ShellCommand +from buildbot.steps.transfer import FileDownload + +f.addStep(FileDownload(mastersrc="~/todays_build_config.txt", + slavedest="build_config.txt")) +f.addStep(ShellCommand(command=["make", "config"])) +@end example + +Like @code{FileUpload}, the @code{mastersrc=} argument is interpreted +relative to the buildmaster's base directory, and the +@code{slavedest=} argument is relative to the builder's working +directory. If the buildslave is running in @file{~buildslave}, and the +builder's ``builddir'' is something like @file{tests-i386}, then the +workdir is going to be @file{~buildslave/tests-i386/build}, and a +@code{slavedest=} of @file{foo/bar.html} will get put in +@file{~buildslave/tests-i386/build/foo/bar.html}. Both of these commands +will create any missing intervening directories. + +@subheading Other Parameters + +The @code{maxsize=} argument lets you set a maximum size for the file +to be transferred. This may help to avoid surprises: transferring a +100MB coredump when you were expecting to move a 10kB status file +might take an awfully long time. The @code{blocksize=} argument +controls how the file is sent over the network: larger blocksizes are +slightly more efficient but also consume more memory on each end, and +there is a hard-coded limit of about 640kB. + +The @code{mode=} argument allows you to control the access permissions +of the target file, traditionally expressed as an octal integer. The +most common value is probably 0755, which sets the ``x'' executable +bit on the file (useful for shell scripts and the like). The default +value for @code{mode=} is None, which means the permission bits will +default to whatever the umask of the writing process is. The default +umask tends to be fairly restrictive, but at least on the buildslave +you can make it less restrictive with a --umask command-line option at +creation time (@pxref{Buildslave Options}). + +@subheading Transfering Directories + +To transfer complete directories from the buildslave to the master, there +is a BuildStep named @code{DirectoryUpload}. It works like @code{FileUpload}, +just for directories. However it does not support the @code{maxsize}, +@code{blocksize} and @code{mode} arguments. As an example, let's assume an +generated project documentation, which consists of many files (like the output +of doxygen or epydoc). We want to move the entire documentation to the +buildmaster, into a @code{~/public_html/docs} directory. On the slave-side +the directory can be found under @code{docs}: + +@example +from buildbot.steps.shell import ShellCommand +from buildbot.steps.transfer import DirectoryUpload + +f.addStep(ShellCommand(command=["make", "docs"])) +f.addStep(DirectoryUpload(slavesrc="docs", + masterdest="~/public_html/docs")) +@end example + +The DirectoryUpload step will create all necessary directories and +transfers empty directories, too. + +@node Steps That Run on the Master +@subsection Steps That Run on the Master + +Occasionally, it is useful to execute some task on the master, for example to +create a directory, deploy a build result, or trigger some other centralized +processing. This is possible, in a limited fashion, with the +@code{MasterShellCommand} step. + +This step operates similarly to a regular @code{ShellCommand}, but executes on +the master, instead of the slave. To be clear, the enclosing @code{Build} +object must still have a slave object, just as for any other step -- only, in +this step, the slave does not do anything. + +In this example, the step renames a tarball based on the day of the week. + +@example +from buildbot.steps.transfer import FileUpload +from buildbot.steps.master import MasterShellCommand + +f.addStep(FileUpload(slavesrc="widgetsoft.tar.gz", + masterdest="/var/buildoutputs/widgetsoft-new.tar.gz")) +f.addStep(MasterShellCommand(command=""" + cd /var/buildoutputs; + mv widgetsoft-new.tar.gz widgetsoft-`date +%a`.tar.gz""")) +@end example + +@node Triggering Schedulers +@subsection Triggering Schedulers + +The counterpart to the Triggerable described in section +@pxref{Triggerable Scheduler} is the Trigger BuildStep. + +@example +from buildbot.steps.trigger import Trigger +f.addStep(Trigger(schedulerNames=['build-prep'], + waitForFinish=True, + updateSourceStamp=True, + set_properties=@{ 'quick' : False @}, + copy_properties=[ 'release_code_name' ])) +@end example + +The @code{schedulerNames=} argument lists the Triggerables +that should be triggered when this step is executed. Note that +it is possible, but not advisable, to create a cycle where a build +continually triggers itself, because the schedulers are specified +by name. + +If @code{waitForFinish} is True, then the step will not finish until +all of the builds from the triggered schedulers have finished. If this +argument is False (the default) or not given, then the buildstep +succeeds immediately after triggering the schedulers. + +If @code{updateSourceStamp} is True (the default), then step updates +the SourceStamp given to the Triggerables to include +@code{got_revision} (the revision actually used in this build) as +@code{revision} (the revision to use in the triggered builds). This is +useful to ensure that all of the builds use exactly the same +SourceStamp, even if other Changes have occurred while the build was +running. + +Two parameters allow control of the properties that are passed to the triggered +scheduler. To simply copy properties verbatim, list them in the +@code{copy_properties} parameter. To set properties explicitly, use the more +sophisticated @code{set_properties}, which takes a dictionary mapping property +names to values. You may use @code{WithProperties} here to dynamically +construct new property values. + +@node Writing New BuildSteps +@subsection Writing New BuildSteps + +While it is a good idea to keep your build process self-contained in +the source code tree, sometimes it is convenient to put more +intelligence into your Buildbot configuration. One way to do this is +to write a custom BuildStep. Once written, this Step can be used in +the @file{master.cfg} file. + +The best reason for writing a custom BuildStep is to better parse the +results of the command being run. For example, a BuildStep that knows +about JUnit could look at the logfiles to determine which tests had +been run, how many passed and how many failed, and then report more +detailed information than a simple @code{rc==0} -based ``good/bad'' +decision. + +@menu +* Writing BuildStep Constructors:: +* BuildStep LogFiles:: +* Reading Logfiles:: +* Adding LogObservers:: +* BuildStep URLs:: +@end menu + +@node Writing BuildStep Constructors +@subsubsection Writing BuildStep Constructors + +BuildStep classes have some extra equipment, because they are their own +factories. Consider the use of a BuildStep in @file{master.cfg}: + +@example +f.addStep(MyStep(someopt="stuff", anotheropt=1)) +@end example + +This creates a single instance of class @code{MyStep}. However, Buildbot needs +a new object each time the step is executed. this is accomplished by storing +the information required to instantiate a new object in the @code{factory} +attribute. When the time comes to construct a new Build, BuildFactory consults +this attribute (via @code{getStepFactory}) and instantiates a new step object. + +When writing a new step class, then, keep in mind are that you cannot do +anything "interesting" in the constructor -- limit yourself to checking and +storing arguments. To ensure that these arguments are provided to any new +objects, call @code{self.addFactoryArguments} with any keyword arguments your +constructor needs. + +Keep a @code{**kwargs} argument on the end of your options, and pass that up to +the parent class's constructor. + +The whole thing looks like this: + +@example +class Frobinfy(LoggingBuildStep): + def __init__(self, + frob_what="frobee", + frob_how_many=None, + frob_how=None, + **kwargs) + + # check + if frob_how_many is None: + raise TypeError("Frobinfy argument how_many is required") + + # call parent + LoggingBuildStep.__init__(self, **kwargs) + + # and record arguments for later + self.addFactoryArguments( + frob_what=frob_what, + frob_how_many=frob_how_many, + frob_how=frob_how) + +class FastFrobnify(Frobnify): + def __init__(self, + speed=5, + **kwargs) + Frobnify.__init__(self, **kwargs) + self.addFactoryArguments( + speed=speed) +@end example + +@node BuildStep LogFiles +@subsubsection BuildStep LogFiles + +Each BuildStep has a collection of ``logfiles''. Each one has a short +name, like ``stdio'' or ``warnings''. Each LogFile contains an +arbitrary amount of text, usually the contents of some output file +generated during a build or test step, or a record of everything that +was printed to stdout/stderr during the execution of some command. + +These LogFiles are stored to disk, so they can be retrieved later. + +Each can contain multiple ``channels'', generally limited to three +basic ones: stdout, stderr, and ``headers''. For example, when a +ShellCommand runs, it writes a few lines to the ``headers'' channel to +indicate the exact argv strings being run, which directory the command +is being executed in, and the contents of the current environment +variables. Then, as the command runs, it adds a lot of ``stdout'' and +``stderr'' messages. When the command finishes, a final ``header'' +line is added with the exit code of the process. + +Status display plugins can format these different channels in +different ways. For example, the web page shows LogFiles as text/html, +with header lines in blue text, stdout in black, and stderr in red. A +different URL is available which provides a text/plain format, in +which stdout and stderr are collapsed together, and header lines are +stripped completely. This latter option makes it easy to save the +results to a file and run @command{grep} or whatever against the +output. + +Each BuildStep contains a mapping (implemented in a python dictionary) +from LogFile name to the actual LogFile objects. Status plugins can +get a list of LogFiles to display, for example, a list of HREF links +that, when clicked, provide the full contents of the LogFile. + +@heading Using LogFiles in custom BuildSteps + +The most common way for a custom BuildStep to use a LogFile is to +summarize the results of a ShellCommand (after the command has +finished running). For example, a compile step with thousands of lines +of output might want to create a summary of just the warning messages. +If you were doing this from a shell, you would use something like: + +@example +grep "warning:" output.log >warnings.log +@end example + +In a custom BuildStep, you could instead create a ``warnings'' LogFile +that contained the same text. To do this, you would add code to your +@code{createSummary} method that pulls lines from the main output log +and creates a new LogFile with the results: + +@example + def createSummary(self, log): + warnings = [] + for line in log.readlines(): + if "warning:" in line: + warnings.append() + self.addCompleteLog('warnings', "".join(warnings)) +@end example + +This example uses the @code{addCompleteLog} method, which creates a +new LogFile, puts some text in it, and then ``closes'' it, meaning +that no further contents will be added. This LogFile will appear in +the HTML display under an HREF with the name ``warnings'', since that +is the name of the LogFile. + +You can also use @code{addHTMLLog} to create a complete (closed) +LogFile that contains HTML instead of plain text. The normal LogFile +will be HTML-escaped if presented through a web page, but the HTML +LogFile will not. At the moment this is only used to present a pretty +HTML representation of an otherwise ugly exception traceback when +something goes badly wrong during the BuildStep. + +In contrast, you might want to create a new LogFile at the beginning +of the step, and add text to it as the command runs. You can create +the LogFile and attach it to the build by calling @code{addLog}, which +returns the LogFile object. You then add text to this LogFile by +calling methods like @code{addStdout} and @code{addHeader}. When you +are done, you must call the @code{finish} method so the LogFile can be +closed. It may be useful to create and populate a LogFile like this +from a LogObserver method @xref{Adding LogObservers}. + +The @code{logfiles=} argument to @code{ShellCommand} (see +@pxref{ShellCommand}) creates new LogFiles and fills them in realtime +by asking the buildslave to watch a actual file on disk. The +buildslave will look for additions in the target file and report them +back to the BuildStep. These additions will be added to the LogFile by +calling @code{addStdout}. These secondary LogFiles can be used as the +source of a LogObserver just like the normal ``stdio'' LogFile. + +@node Reading Logfiles +@subsubsection Reading Logfiles + +Once a LogFile has been added to a BuildStep with @code{addLog()}, +@code{addCompleteLog()}, @code{addHTMLLog()}, or @code{logfiles=}, +your BuildStep can retrieve it by using @code{getLog()}: + +@example +class MyBuildStep(ShellCommand): + logfiles = @{ "nodelog": "_test/node.log" @} + + def evaluateCommand(self, cmd): + nodelog = self.getLog("nodelog") + if "STARTED" in nodelog.getText(): + return SUCCESS + else: + return FAILURE +@end example + +For a complete list of the methods you can call on a LogFile, please +see the docstrings on the @code{IStatusLog} class in +@file{buildbot/interfaces.py}. + + +@node Adding LogObservers +@subsubsection Adding LogObservers + +@cindex LogObserver +@cindex LogLineObserver + +Most shell commands emit messages to stdout or stderr as they operate, +especially if you ask them nicely with a @code{--verbose} flag of some +sort. They may also write text to a log file while they run. Your +BuildStep can watch this output as it arrives, to keep track of how +much progress the command has made. You can get a better measure of +progress by counting the number of source files compiled or test cases +run than by merely tracking the number of bytes that have been written +to stdout. This improves the accuracy and the smoothness of the ETA +display. + +To accomplish this, you will need to attach a @code{LogObserver} to +one of the log channels, most commonly to the ``stdio'' channel but +perhaps to another one which tracks a log file. This observer is given +all text as it is emitted from the command, and has the opportunity to +parse that output incrementally. Once the observer has decided that +some event has occurred (like a source file being compiled), it can +use the @code{setProgress} method to tell the BuildStep about the +progress that this event represents. + +There are a number of pre-built @code{LogObserver} classes that you +can choose from (defined in @code{buildbot.process.buildstep}, and of +course you can subclass them to add further customization. The +@code{LogLineObserver} class handles the grunt work of buffering and +scanning for end-of-line delimiters, allowing your parser to operate +on complete stdout/stderr lines. (Lines longer than a set maximum +length are dropped; the maximum defaults to 16384 bytes, but you can +change it by calling @code{setMaxLineLength()} on your +@code{LogLineObserver} instance. Use @code{sys.maxint} for effective +infinity.) + +For example, let's take a look at the @code{TrialTestCaseCounter}, +which is used by the Trial step to count test cases as they are run. +As Trial executes, it emits lines like the following: + +@example +buildbot.test.test_config.ConfigTest.testDebugPassword ... [OK] +buildbot.test.test_config.ConfigTest.testEmpty ... [OK] +buildbot.test.test_config.ConfigTest.testIRC ... [FAIL] +buildbot.test.test_config.ConfigTest.testLocks ... [OK] +@end example + +When the tests are finished, trial emits a long line of ``======'' and +then some lines which summarize the tests that failed. We want to +avoid parsing these trailing lines, because their format is less +well-defined than the ``[OK]'' lines. + +The parser class looks like this: + +@example +from buildbot.process.buildstep import LogLineObserver + +class TrialTestCaseCounter(LogLineObserver): + _line_re = re.compile(r'^([\w\.]+) \.\.\. \[([^\]]+)\]$') + numTests = 0 + finished = False + + def outLineReceived(self, line): + if self.finished: + return + if line.startswith("=" * 40): + self.finished = True + return + + m = self._line_re.search(line.strip()) + if m: + testname, result = m.groups() + self.numTests += 1 + self.step.setProgress('tests', self.numTests) +@end example + +This parser only pays attention to stdout, since that's where trial +writes the progress lines. It has a mode flag named @code{finished} to +ignore everything after the ``===='' marker, and a scary-looking +regular expression to match each line while hopefully ignoring other +messages that might get displayed as the test runs. + +Each time it identifies a test has been completed, it increments its +counter and delivers the new progress value to the step with +@code{self.step.setProgress}. This class is specifically measuring +progress along the ``tests'' metric, in units of test cases (as +opposed to other kinds of progress like the ``output'' metric, which +measures in units of bytes). The Progress-tracking code uses each +progress metric separately to come up with an overall completion +percentage and an ETA value. + +To connect this parser into the @code{Trial} BuildStep, +@code{Trial.__init__} ends with the following clause: + +@example + # this counter will feed Progress along the 'test cases' metric + counter = TrialTestCaseCounter() + self.addLogObserver('stdio', counter) + self.progressMetrics += ('tests',) +@end example + +This creates a TrialTestCaseCounter and tells the step that the +counter wants to watch the ``stdio'' log. The observer is +automatically given a reference to the step in its @code{.step} +attribute. + +@subheading A Somewhat Whimsical Example + +Let's say that we've got some snazzy new unit-test framework called +Framboozle. It's the hottest thing since sliced bread. It slices, it +dices, it runs unit tests like there's no tomorrow. Plus if your unit +tests fail, you can use its name for a Web 2.1 startup company, make +millions of dollars, and hire engineers to fix the bugs for you, while +you spend your afternoons lazily hang-gliding along a scenic pacific +beach, blissfully unconcerned about the state of your +tests.@footnote{framboozle.com is still available. Remember, I get 10% +:).} + +To run a Framboozle-enabled test suite, you just run the 'framboozler' +command from the top of your source code tree. The 'framboozler' +command emits a bunch of stuff to stdout, but the most interesting bit +is that it emits the line "FNURRRGH!" every time it finishes running a +test case@footnote{Framboozle gets very excited about running unit +tests.}. You'd like to have a test-case counting LogObserver that +watches for these lines and counts them, because counting them will +help the buildbot more accurately calculate how long the build will +take, and this will let you know exactly how long you can sneak out of +the office for your hang-gliding lessons without anyone noticing that +you're gone. + +This will involve writing a new BuildStep (probably named +"Framboozle") which inherits from ShellCommand. The BuildStep class +definition itself will look something like this: + +@example +# START +from buildbot.steps.shell import ShellCommand +from buildbot.process.buildstep import LogLineObserver + +class FNURRRGHCounter(LogLineObserver): + numTests = 0 + def outLineReceived(self, line): + if "FNURRRGH!" in line: + self.numTests += 1 + self.step.setProgress('tests', self.numTests) + +class Framboozle(ShellCommand): + command = ["framboozler"] + + def __init__(self, **kwargs): + ShellCommand.__init__(self, **kwargs) # always upcall! + counter = FNURRRGHCounter()) + self.addLogObserver('stdio', counter) + self.progressMetrics += ('tests',) +# FINISH +@end example + +So that's the code that we want to wind up using. How do we actually +deploy it? + +You have a couple of different options. + +Option 1: The simplest technique is to simply put this text +(everything from START to FINISH) in your master.cfg file, somewhere +before the BuildFactory definition where you actually use it in a +clause like: + +@example +f = BuildFactory() +f.addStep(SVN(svnurl="stuff")) +f.addStep(Framboozle()) +@end example + +Remember that master.cfg is secretly just a python program with one +job: populating the BuildmasterConfig dictionary. And python programs +are allowed to define as many classes as they like. So you can define +classes and use them in the same file, just as long as the class is +defined before some other code tries to use it. + +This is easy, and it keeps the point of definition very close to the +point of use, and whoever replaces you after that unfortunate +hang-gliding accident will appreciate being able to easily figure out +what the heck this stupid "Framboozle" step is doing anyways. The +downside is that every time you reload the config file, the Framboozle +class will get redefined, which means that the buildmaster will think +that you've reconfigured all the Builders that use it, even though +nothing changed. Bleh. + +Option 2: Instead, we can put this code in a separate file, and import +it into the master.cfg file just like we would the normal buildsteps +like ShellCommand and SVN. + +Create a directory named ~/lib/python, put everything from START to +FINISH in ~/lib/python/framboozle.py, and run your buildmaster using: + +@example + PYTHONPATH=~/lib/python buildbot start MASTERDIR +@end example + +or use the @file{Makefile.buildbot} to control the way +@command{buildbot start} works. Or add something like this to +something like your ~/.bashrc or ~/.bash_profile or ~/.cshrc: + +@example + export PYTHONPATH=~/lib/python +@end example + +Once we've done this, our master.cfg can look like: + +@example +from framboozle import Framboozle +f = BuildFactory() +f.addStep(SVN(svnurl="stuff")) +f.addStep(Framboozle()) +@end example + +or: + +@example +import framboozle +f = BuildFactory() +f.addStep(SVN(svnurl="stuff")) +f.addStep(framboozle.Framboozle()) +@end example + +(check out the python docs for details about how "import" and "from A +import B" work). + +What we've done here is to tell python that every time it handles an +"import" statement for some named module, it should look in our +~/lib/python/ for that module before it looks anywhere else. After our +directories, it will try in a bunch of standard directories too +(including the one where buildbot is installed). By setting the +PYTHONPATH environment variable, you can add directories to the front +of this search list. + +Python knows that once it "import"s a file, it doesn't need to +re-import it again. This means that reconfiguring the buildmaster +(with "buildbot reconfig", for example) won't make it think the +Framboozle class has changed every time, so the Builders that use it +will not be spuriously restarted. On the other hand, you either have +to start your buildmaster in a slightly weird way, or you have to +modify your environment to set the PYTHONPATH variable. + + +Option 3: Install this code into a standard python library directory + +Find out what your python's standard include path is by asking it: + +@example +80:warner@@luther% python +Python 2.4.4c0 (#2, Oct 2 2006, 00:57:46) +[GCC 4.1.2 20060928 (prerelease) (Debian 4.1.1-15)] on linux2 +Type "help", "copyright", "credits" or "license" for more information. +>>> import sys +>>> import pprint +>>> pprint.pprint(sys.path) +['', + '/usr/lib/python24.zip', + '/usr/lib/python2.4', + '/usr/lib/python2.4/plat-linux2', + '/usr/lib/python2.4/lib-tk', + '/usr/lib/python2.4/lib-dynload', + '/usr/local/lib/python2.4/site-packages', + '/usr/lib/python2.4/site-packages', + '/usr/lib/python2.4/site-packages/Numeric', + '/var/lib/python-support/python2.4', + '/usr/lib/site-python'] +@end example + +In this case, putting the code into +/usr/local/lib/python2.4/site-packages/framboozle.py would work just +fine. We can use the same master.cfg "import framboozle" statement as +in Option 2. By putting it in a standard include directory (instead of +the decidedly non-standard ~/lib/python), we don't even have to set +PYTHONPATH to anything special. The downside is that you probably have +to be root to write to one of those standard include directories. + + +Option 4: Submit the code for inclusion in the Buildbot distribution + +Make a fork of buildbot on http://github.com/djmitche/buildbot or post a patch +in a bug at http://buildbot.net. In either case, post a note about your patch +to the mailing list, so others can provide feedback and, eventually, commit it. + +@example +from buildbot.steps import framboozle +f = BuildFactory() +f.addStep(SVN(svnurl="stuff")) +f.addStep(framboozle.Framboozle()) +@end example + +And then you don't even have to install framboozle.py anywhere on your +system, since it will ship with Buildbot. You don't have to be root, +you don't have to set PYTHONPATH. But you do have to make a good case +for Framboozle being worth going into the main distribution, you'll +probably have to provide docs and some unit test cases, you'll need to +figure out what kind of beer the author likes, and then you'll have to +wait until the next release. But in some environments, all this is +easier than getting root on your buildmaster box, so the tradeoffs may +actually be worth it. + + + +Putting the code in master.cfg (1) makes it available to that +buildmaster instance. Putting it in a file in a personal library +directory (2) makes it available for any buildmasters you might be +running. Putting it in a file in a system-wide shared library +directory (3) makes it available for any buildmasters that anyone on +that system might be running. Getting it into the buildbot's upstream +repository (4) makes it available for any buildmasters that anyone in +the world might be running. It's all a matter of how widely you want +to deploy that new class. + + + +@node BuildStep URLs +@subsubsection BuildStep URLs + +@cindex links +@cindex BuildStep URLs +@cindex addURL + +Each BuildStep has a collection of ``links''. Like its collection of +LogFiles, each link has a name and a target URL. The web status page +creates HREFs for each link in the same box as it does for LogFiles, +except that the target of the link is the external URL instead of an +internal link to a page that shows the contents of the LogFile. + +These external links can be used to point at build information hosted +on other servers. For example, the test process might produce an +intricate description of which tests passed and failed, or some sort +of code coverage data in HTML form, or a PNG or GIF image with a graph +of memory usage over time. The external link can provide an easy way +for users to navigate from the buildbot's status page to these +external web sites or file servers. Note that the step itself is +responsible for insuring that there will be a document available at +the given URL (perhaps by using @command{scp} to copy the HTML output +to a @file{~/public_html/} directory on a remote web server). Calling +@code{addURL} does not magically populate a web server. + +To set one of these links, the BuildStep should call the @code{addURL} +method with the name of the link and the target URL. Multiple URLs can +be set. + +In this example, we assume that the @command{make test} command causes +a collection of HTML files to be created and put somewhere on the +coverage.example.org web server, in a filename that incorporates the +build number. + +@example +class TestWithCodeCoverage(BuildStep): + command = ["make", "test", + WithProperties("buildnum=%s", "buildnumber")] + + def createSummary(self, log): + buildnumber = self.getProperty("buildnumber") + url = "http://coverage.example.org/builds/%s.html" % buildnumber + self.addURL("coverage", url) +@end example + +You might also want to extract the URL from some special message +output by the build process itself: + +@example +class TestWithCodeCoverage(BuildStep): + command = ["make", "test", + WithProperties("buildnum=%s", "buildnumber")] + + def createSummary(self, log): + output = StringIO(log.getText()) + for line in output.readlines(): + if line.startswith("coverage-url:"): + url = line[len("coverage-url:"):].strip() + self.addURL("coverage", url) + return +@end example + +Note that a build process which emits both stdout and stderr might +cause this line to be split or interleaved between other lines. It +might be necessary to restrict the getText() call to only stdout with +something like this: + +@example + output = StringIO("".join([c[1] + for c in log.getChunks() + if c[0] == LOG_CHANNEL_STDOUT])) +@end example + +Of course if the build is run under a PTY, then stdout and stderr will +be merged before the buildbot ever sees them, so such interleaving +will be unavoidable. + + +@node Interlocks +@section Interlocks + +@cindex locks +@slindex buildbot.locks.MasterLock +@slindex buildbot.locks.SlaveLock +@slindex buildbot.locks.LockAccess + +Until now, we assumed that a master can run builds at any slave whenever +needed or desired. Some times, you want to enforce additional constraints on +builds. For reasons like limited network bandwidth, old slave machines, or a +self-willed data base server, you may want to limit the number of builds (or +build steps) that can access a resource. + +The mechanism used by Buildbot is known as the read/write lock.@footnote{See +http://en.wikipedia.org/wiki/Read/write_lock_pattern for more information.} It +allows either many readers or a single writer but not a combination of readers +and writers. The general lock has been modified and extended for use in +Buildbot. Firstly, the general lock allows an infinite number of readers. In +Buildbot, we often want to put an upper limit on the number of readers, for +example allowing two out of five possible builds at the same time. To do this, +the lock counts the number of active readers. Secondly, the terms @emph{read +mode} and @emph{write mode} are confusing in Buildbot context. They have been +replaced by @emph{counting mode} (since the lock counts them) and @emph{exclusive +mode}. As a result of these changes, locks in Buildbot allow a number of +builds (upto some fixed number) in counting mode, or they allow one build in +exclusive mode. + +Often, not all slaves are equal. To allow for this situation, Buildbot allows +to have a separate upper limit on the count for each slave. In this way, you +can have at most 3 concurrent builds at a fast slave, 2 at a slightly older +slave, and 1 at all other slaves. + +The final thing you can specify when you introduce a new lock is its scope. +Some constraints are global -- they must be enforced over all slaves. Other +constraints are local to each slave. A @emph{master lock} is used for the +global constraints. You can ensure for example that at most one build (of all +builds running at all slaves) accesses the data base server. With a +@emph{slave lock} you can add a limit local to each slave. With such a lock, +you can for example enforce an upper limit to the number of active builds at a +slave, like above. + +Time for a few examples. Below a master lock is defined to protect a data base, +and a slave lock is created to limit the number of builds at each slave. + +@example +from buildbot import locks + +db_lock = locks.MasterLock("database") +build_lock = locks.SlaveLock("slave_builds", + maxCount = 1, + maxCountForSlave = @{ 'fast': 3, 'new': 2 @}) +@end example + +After importing locks from buildbot, @code{db_lock} is defined to be a master +lock. The @code{"database"} string is used for uniquely identifying the lock. +At the next line, a slave lock called @code{build_lock} is created. It is +identified by the @code{"slave_builds"} string. Since the requirements of the +lock are a bit more complicated, two optional arguments are also specified. The +@code{maxCount} parameter sets the default limit for builds in counting mode to +@code{1}. For the slave called @code{'fast'} however, we want to have at most +three builds, and for the slave called @code{'new'} the upper limit is two +builds running at the same time. + +The next step is using the locks in builds. Buildbot allows a lock to be used +during an entire build (from beginning to end), or only during a single build +step. In the latter case, the lock is claimed for use just before the step +starts, and released again when the step ends. To prevent +deadlocks,@footnote{Deadlock is the situation where two or more slaves each +hold a lock in exclusive mode, and in addition want to claim the lock held by +the other slave exclusively as well. Since locks allow at most one exclusive +user, both slaves will wait forever.} it is not possible to claim or release +locks at other times. + +To use locks, you should add them with a @code{locks} argument. +Each use of a lock is either in counting mode (that is, possibly shared with +other builds) or in exclusive mode. A build or build step proceeds only when it +has acquired all locks. If a build or step needs a lot of locks, it may be +starved@footnote{Starving is the situation that only a few locks are available, +and they are immediately grabbed by another build. As a result, it may take a +long time before all locks needed by the starved build are free at the same +time.} by other builds that need fewer locks. + +To illustrate use of locks, a few examples. + +@example +from buildbot import locks +from buildbot.steps import source, shell +from buildbot.process import factory + +db_lock = locks.MasterLock("database") +build_lock = locks.SlaveLock("slave_builds", + maxCount = 1, + maxCountForSlave = @{ 'fast': 3, 'new': 2 @}) + +f = factory.BuildFactory() +f.addStep(source.SVN(svnurl="http://example.org/svn/Trunk")) +f.addStep(shell.ShellCommand(command="make all")) +f.addStep(shell.ShellCommand(command="make test", + locks=[db_lock.access('exclusive')])) + +b1 = @{'name': 'full1', 'slavename': 'fast', 'builddir': 'f1', 'factory': f, + 'locks': [build_lock.access('counting')] @} + +b2 = @{'name': 'full2', 'slavename': 'new', 'builddir': 'f2', 'factory': f. + 'locks': [build_lock.access('counting')] @} + +b3 = @{'name': 'full3', 'slavename': 'old', 'builddir': 'f3', 'factory': f. + 'locks': [build_lock.access('counting')] @} + +b4 = @{'name': 'full4', 'slavename': 'other', 'builddir': 'f4', 'factory': f. + 'locks': [build_lock.access('counting')] @} + +c['builders'] = [b1, b2, b3, b4] +@end example + +Here we have four slaves @code{b1}, @code{b2}, @code{b3}, and @code{b4}. Each +slave performs the same checkout, make, and test build step sequence. +We want to enforce that at most one test step is executed between all slaves due +to restrictions with the data base server. This is done by adding the +@code{locks=} parameter with the third step. It takes a list of locks with their +access mode. In this case only the @code{db_lock} is needed. The exclusive +access mode is used to ensure there is at most one slave that executes the test +step. + +In addition to exclusive accessing the data base, we also want slaves to stay +responsive even under the load of a large number of builds being triggered. +For this purpose, the slave lock called @code{build_lock} is defined. Since +the restraint holds for entire builds, the lock is specified in the builder +with @code{'locks': [build_lock.access('counting')]}. +@node Build Factories +@section Build Factories + + +Each Builder is equipped with a ``build factory'', which is +responsible for producing the actual @code{Build} objects that perform +each build. This factory is created in the configuration file, and +attached to a Builder through the @code{factory} element of its +dictionary. + +The standard @code{BuildFactory} object creates @code{Build} objects +by default. These Builds will each execute a collection of BuildSteps +in a fixed sequence. Each step can affect the results of the build, +but in general there is little intelligence to tie the different steps +together. You can create subclasses of @code{Build} to implement more +sophisticated build processes, and then use a subclass of +@code{BuildFactory} (or simply set the @code{buildClass} attribute) to +create instances of your new Build subclass. + + +@menu +* BuildStep Objects:: +* BuildFactory:: +* Process-Specific build factories:: +@end menu + +@node BuildStep Objects +@subsection BuildStep Objects + +The steps used by these builds are all subclasses of @code{BuildStep}. +The standard ones provided with Buildbot are documented later, +@xref{Build Steps}. You can also write your own subclasses to use in +builds. + +The basic behavior for a @code{BuildStep} is to: + +@itemize @bullet +@item +run for a while, then stop +@item +possibly invoke some RemoteCommands on the attached build slave +@item +possibly produce a set of log files +@item +finish with a status described by one of four values defined in +buildbot.status.builder: SUCCESS, WARNINGS, FAILURE, SKIPPED +@item +provide a list of short strings to describe the step +@item +define a color (generally green, orange, or red) with which the +step should be displayed +@end itemize + + +More sophisticated steps may produce additional information and +provide it to later build steps, or store it in the factory to provide +to later builds. + + +@menu +* BuildFactory Attributes:: +* Quick builds:: +@end menu + +@node BuildFactory +@subsection BuildFactory + +@bfindex buildbot.process.factory.BuildFactory +@bfindex buildbot.process.factory.BasicBuildFactory +@c TODO: what is BasicSVN anyway? +@bfindex buildbot.process.factory.BasicSVN + +The default @code{BuildFactory}, provided in the +@code{buildbot.process.factory} module, contains an internal list of +``BuildStep specifications'': a list of @code{(step_class, kwargs)} +tuples for each. These specification tuples are constructed when the +config file is read, by asking the instances passed to @code{addStep} +for their subclass and arguments. + +When asked to create a Build, the @code{BuildFactory} puts a copy of +the list of step specifications into the new Build object. When the +Build is actually started, these step specifications are used to +create the actual set of BuildSteps, which are then executed one at a +time. This serves to give each Build an independent copy of each step. +For example, a build which consists of a CVS checkout followed by a +@code{make build} would be constructed as follows: + +@example +from buildbot.steps import source, shell +from buildbot.process import factory + +f = factory.BuildFactory() +f.addStep(source.CVS(cvsroot=CVSROOT, cvsmodule="project", mode="update")) +f.addStep(shell.Compile(command=["make", "build"])) +@end example + +(To support config files from buildbot-0.7.5 and earlier, +@code{addStep} also accepts the @code{f.addStep(shell.Compile, +command=["make","build"])} form, although its use is discouraged +because then the @code{Compile} step doesn't get to validate or +complain about its arguments until build time. The modern +pass-by-instance approach allows this validation to occur while the +config file is being loaded, where the admin has a better chance of +noticing problems). + +It is also possible to pass a list of steps into the +@code{BuildFactory} when it is created. Using @code{addStep} is +usually simpler, but there are cases where is is more convenient to +create the list of steps ahead of time.: + +@example +from buildbot.steps import source, shell +from buildbot.process import factory + +all_steps = [source.CVS(cvsroot=CVSROOT, cvsmodule="project", mode="update"), + shell.Compile(command=["make", "build"]), + ] +f = factory.BuildFactory(all_steps) +@end example + + +Each step can affect the build process in the following ways: + +@itemize @bullet +@item +If the step's @code{haltOnFailure} attribute is True, then a failure +in the step (i.e. if it completes with a result of FAILURE) will cause +the whole build to be terminated immediately: no further steps will be +executed, with the exception of steps with @code{alwaysRun} set to +True. @code{haltOnFailure} is useful for setup steps upon which the +rest of the build depends: if the CVS checkout or @code{./configure} +process fails, there is no point in trying to compile or test the +resulting tree. + +@item +If the step's @code{alwaysRun} attribute is True, then it will always +be run, regardless of if previous steps have failed. This is useful +for cleanup steps that should always be run to return the build +directory or build slave into a good state. + +@item +If the @code{flunkOnFailure} or @code{flunkOnWarnings} flag is set, +then a result of FAILURE or WARNINGS will mark the build as a whole as +FAILED. However, the remaining steps will still be executed. This is +appropriate for things like multiple testing steps: a failure in any +one of them will indicate that the build has failed, however it is +still useful to run them all to completion. + +@item +Similarly, if the @code{warnOnFailure} or @code{warnOnWarnings} flag +is set, then a result of FAILURE or WARNINGS will mark the build as +having WARNINGS, and the remaining steps will still be executed. This +may be appropriate for certain kinds of optional build or test steps. +For example, a failure experienced while building documentation files +should be made visible with a WARNINGS result but not be serious +enough to warrant marking the whole build with a FAILURE. + +@end itemize + +In addition, each Step produces its own results, may create logfiles, +etc. However only the flags described above have any effect on the +build as a whole. + +The pre-defined BuildSteps like @code{CVS} and @code{Compile} have +reasonably appropriate flags set on them already. For example, without +a source tree there is no point in continuing the build, so the +@code{CVS} class has the @code{haltOnFailure} flag set to True. Look +in @file{buildbot/steps/*.py} to see how the other Steps are +marked. + +Each Step is created with an additional @code{workdir} argument that +indicates where its actions should take place. This is specified as a +subdirectory of the slave builder's base directory, with a default +value of @code{build}. This is only implemented as a step argument (as +opposed to simply being a part of the base directory) because the +CVS/SVN steps need to perform their checkouts from the parent +directory. + +@menu +* BuildFactory Attributes:: +* Quick builds:: +@end menu + +@node BuildFactory Attributes +@subsubsection BuildFactory Attributes + +Some attributes from the BuildFactory are copied into each Build. + +@cindex treeStableTimer + +@table @code +@item useProgress +(defaults to True): if True, the buildmaster keeps track of how long +each step takes, so it can provide estimates of how long future builds +will take. If builds are not expected to take a consistent amount of +time (such as incremental builds in which a random set of files are +recompiled or tested each time), this should be set to False to +inhibit progress-tracking. + +@end table + + +@node Quick builds +@subsubsection Quick builds + +@bfindex buildbot.process.factory.QuickBuildFactory + +The difference between a ``full build'' and a ``quick build'' is that +quick builds are generally done incrementally, starting with the tree +where the previous build was performed. That simply means that the +source-checkout step should be given a @code{mode='update'} flag, to +do the source update in-place. + +In addition to that, the @code{useProgress} flag should be set to +False. Incremental builds will (or at least the ought to) compile as +few files as necessary, so they will take an unpredictable amount of +time to run. Therefore it would be misleading to claim to predict how +long the build will take. + + +@node Process-Specific build factories +@subsection Process-Specific build factories + +Many projects use one of a few popular build frameworks to simplify +the creation and maintenance of Makefiles or other compilation +structures. Buildbot provides several pre-configured BuildFactory +subclasses which let you build these projects with a minimum of fuss. + +@menu +* GNUAutoconf:: +* CPAN:: +* Python distutils:: +* Python/Twisted/trial projects:: +@end menu + +@node GNUAutoconf +@subsubsection GNUAutoconf + +@bfindex buildbot.process.factory.GNUAutoconf + +@uref{http://www.gnu.org/software/autoconf/, GNU Autoconf} is a +software portability tool, intended to make it possible to write +programs in C (and other languages) which will run on a variety of +UNIX-like systems. Most GNU software is built using autoconf. It is +frequently used in combination with GNU automake. These tools both +encourage a build process which usually looks like this: + +@example +% CONFIG_ENV=foo ./configure --with-flags +% make all +% make check +# make install +@end example + +(except of course the Buildbot always skips the @code{make install} +part). + +The Buildbot's @code{buildbot.process.factory.GNUAutoconf} factory is +designed to build projects which use GNU autoconf and/or automake. The +configuration environment variables, the configure flags, and command +lines used for the compile and test are all configurable, in general +the default values will be suitable. + +Example: + +@example +# use the s() convenience function defined earlier +f = factory.GNUAutoconf(source=s(source.SVN, svnurl=URL, mode="copy"), + flags=["--disable-nls"]) +@end example + +Required Arguments: + +@table @code +@item source +This argument must be a step specification tuple that provides a +BuildStep to generate the source tree. +@end table + +Optional Arguments: + +@table @code +@item configure +The command used to configure the tree. Defaults to +@code{./configure}. Accepts either a string or a list of shell argv +elements. + +@item configureEnv +The environment used for the initial configuration step. This accepts +a dictionary which will be merged into the buildslave's normal +environment. This is commonly used to provide things like +@code{CFLAGS="-O2 -g"} (to turn off debug symbols during the compile). +Defaults to an empty dictionary. + +@item configureFlags +A list of flags to be appended to the argument list of the configure +command. This is commonly used to enable or disable specific features +of the autoconf-controlled package, like @code{["--without-x"]} to +disable windowing support. Defaults to an empty list. + +@item compile +this is a shell command or list of argv values which is used to +actually compile the tree. It defaults to @code{make all}. If set to +None, the compile step is skipped. + +@item test +this is a shell command or list of argv values which is used to run +the tree's self-tests. It defaults to @code{make check}. If set to +None, the test step is skipped. + +@end table + + +@node CPAN +@subsubsection CPAN + +@bfindex buildbot.process.factory.CPAN + +Most Perl modules available from the @uref{http://www.cpan.org/, CPAN} +archive use the @code{MakeMaker} module to provide configuration, +build, and test services. The standard build routine for these modules +looks like: + +@example +% perl Makefile.PL +% make +% make test +# make install +@end example + +(except again Buildbot skips the install step) + +Buildbot provides a @code{CPAN} factory to compile and test these +projects. + + +Arguments: +@table @code +@item source +(required): A step specification tuple, like that used by GNUAutoconf. + +@item perl +A string which specifies the @code{perl} executable to use. Defaults +to just @code{perl}. + +@end table + + +@node Python distutils +@subsubsection Python distutils + +@bfindex buildbot.process.factory.Distutils + +Most Python modules use the @code{distutils} package to provide +configuration and build services. The standard build process looks +like: + +@example +% python ./setup.py build +% python ./setup.py install +@end example + +Unfortunately, although Python provides a standard unit-test framework +named @code{unittest}, to the best of my knowledge @code{distutils} +does not provide a standardized target to run such unit tests. (Please +let me know if I'm wrong, and I will update this factory.) + +The @code{Distutils} factory provides support for running the build +part of this process. It accepts the same @code{source=} parameter as +the other build factories. + + +Arguments: +@table @code +@item source +(required): A step specification tuple, like that used by GNUAutoconf. + +@item python +A string which specifies the @code{python} executable to use. Defaults +to just @code{python}. + +@item test +Provides a shell command which runs unit tests. This accepts either a +string or a list. The default value is None, which disables the test +step (since there is no common default command to run unit tests in +distutils modules). + +@end table + + +@node Python/Twisted/trial projects +@subsubsection Python/Twisted/trial projects + +@bfindex buildbot.process.factory.Trial +@c TODO: document these steps better +@bsindex buildbot.steps.python_twisted.HLint +@bsindex buildbot.steps.python_twisted.Trial +@bsindex buildbot.steps.python_twisted.ProcessDocs +@bsindex buildbot.steps.python_twisted.BuildDebs +@bsindex buildbot.steps.python_twisted.RemovePYCs + +Twisted provides a unit test tool named @code{trial} which provides a +few improvements over Python's built-in @code{unittest} module. Many +python projects which use Twisted for their networking or application +services also use trial for their unit tests. These modules are +usually built and tested with something like the following: + +@example +% python ./setup.py build +% PYTHONPATH=build/lib.linux-i686-2.3 trial -v PROJECTNAME.test +% python ./setup.py install +@end example + +Unfortunately, the @file{build/lib} directory into which the +built/copied .py files are placed is actually architecture-dependent, +and I do not yet know of a simple way to calculate its value. For many +projects it is sufficient to import their libraries ``in place'' from +the tree's base directory (@code{PYTHONPATH=.}). + +In addition, the @var{PROJECTNAME} value where the test files are +located is project-dependent: it is usually just the project's +top-level library directory, as common practice suggests the unit test +files are put in the @code{test} sub-module. This value cannot be +guessed, the @code{Trial} class must be told where to find the test +files. + +The @code{Trial} class provides support for building and testing +projects which use distutils and trial. If the test module name is +specified, trial will be invoked. The library path used for testing +can also be set. + +One advantage of trial is that the Buildbot happens to know how to +parse trial output, letting it identify which tests passed and which +ones failed. The Buildbot can then provide fine-grained reports about +how many tests have failed, when individual tests fail when they had +been passing previously, etc. + +Another feature of trial is that you can give it a series of source +.py files, and it will search them for special @code{test-case-name} +tags that indicate which test cases provide coverage for that file. +Trial can then run just the appropriate tests. This is useful for +quick builds, where you want to only run the test cases that cover the +changed functionality. + +Arguments: +@table @code +@item testpath +Provides a directory to add to @code{PYTHONPATH} when running the unit +tests, if tests are being run. Defaults to @code{.} to include the +project files in-place. The generated build library is frequently +architecture-dependent, but may simply be @file{build/lib} for +pure-python modules. + +@item python +which python executable to use. This list will form the start of +the argv array that will launch trial. If you use this, +you should set @code{trial} to an explicit path (like +@code{/usr/bin/trial} or @code{./bin/trial}). The parameter defaults to @code{None}, which +leaves it out entirely (running @code{trial args} instead of +@code{python ./bin/trial args}). Likely values are @code{['python']}, +@code{['python2.2']}, or @code{['python', '-Wall']}. + +@item trial +provides the name of the @code{trial} command. It is occasionally +useful to use an alternate executable, such as @code{trial2.2} which +might run the tests under an older version of Python. Defaults to +@code{trial}. + +@item trialMode +a list of arguments to pass to trial, specifically to set the reporting mode. +This defaults to @code{['--reporter=bwverbose']}, which only works for +Twisted-2.1.0 and later. + +@item trialArgs +a list of arguments to pass to trial, available to turn on any extra flags you +like. Defaults to @code{[]}. + +@item tests +Provides a module name or names which contain the unit tests for this +project. Accepts a string, typically @code{PROJECTNAME.test}, or a +list of strings. Defaults to None, indicating that no tests should be +run. You must either set this or @code{testChanges}. + +@item testChanges +if True, ignore the @code{tests} parameter and instead ask the Build for all +the files that make up the Changes going into this build. Pass these filenames +to trial and ask it to look for test-case-name tags, running just the tests +necessary to cover the changes. + +@item recurse +If @code{True}, tells Trial (with the @code{--recurse} argument) to look in all +subdirectories for additional test cases. + +@item reactor +which reactor to use, like 'gtk' or 'java'. If not provided, the Twisted's +usual platform-dependent default is used. + +@item randomly +If @code{True}, tells Trial (with the @code{--random=0} argument) to +run the test cases in random order, which sometimes catches subtle +inter-test dependency bugs. Defaults to @code{False}. +@end table + +The step can also take any of the @code{ShellCommand} arguments, e.g., +@code{haltOnFailure}. + +Unless one of @code{tests} or @code{testChanges} are set, the step will +generate an exception. + + diff --git a/docs/changesources.texinfo b/docs/changesources.texinfo new file mode 100644 index 00000000000..af842400d9c --- /dev/null +++ b/docs/changesources.texinfo @@ -0,0 +1,1139 @@ +The most common way to use the Buildbot is centered around the idea of +@code{Source Trees}: a directory tree filled with source code of some form +which can be compiled and/or tested. Some projects use languages that don't +involve any compilation step: nevertheless there may be a @code{build} phase +where files are copied or rearranged into a form that is suitable for +installation. Some projects do not have unit tests, and the Buildbot is +merely helping to make sure that the sources can compile correctly. But in +all of these cases, the thing-being-tested is a single source tree. + +A Version Control System mantains a source tree, and tells the +buildmaster when it changes. The first step of each Build is typically +to acquire a copy of some version of this tree. + +This chapter describes how the Buildbot learns about what Changes have +occurred. For more information on VC systems and Changes, see +@ref{Version Control Systems}. + + +@menu +* Change Sources:: +* Choosing ChangeSources:: +* CVSToys - PBService:: +* Mail-parsing ChangeSources:: +* PBChangeSource:: +* P4Source:: +* BonsaiPoller:: +* SVNPoller:: +* MercurialHook:: +* Bzr Hook:: +* Bzr Poller:: +@end menu + + + +@node Change Sources +@section Change Sources + +@c TODO: rework this, the one-buildmaster-one-tree thing isn't quite +@c so narrow-minded anymore + +Each Buildmaster watches a single source tree. Changes can be provided +by a variety of ChangeSource types, however any given project will +typically have only a single ChangeSource active. This section +provides a description of all available ChangeSource types and +explains how to set up each of them. + +There are a variety of ChangeSources available, some of which are +meant to be used in conjunction with other tools to deliver Change +events from the VC repository to the buildmaster. + +@itemize @bullet + +@item CVSToys +This ChangeSource opens a TCP connection from the buildmaster to a +waiting FreshCVS daemon that lives on the repository machine, and +subscribes to hear about Changes. + +@item MaildirSource +This one watches a local maildir-format inbox for email sent out by +the repository when a change is made. When a message arrives, it is +parsed to create the Change object. A variety of parsing functions are +available to accomodate different email-sending tools. + +@item PBChangeSource +This ChangeSource listens on a local TCP socket for inbound +connections from a separate tool. Usually, this tool would be run on +the VC repository machine in a commit hook. It is expected to connect +to the TCP socket and send a Change message over the network +connection. The @command{buildbot sendchange} command is one example +of a tool that knows how to send these messages, so you can write a +commit script for your VC system that calls it to deliver the Change. +There are other tools in the contrib/ directory that use the same +protocol. + +@end itemize + +As a quick guide, here is a list of VC systems and the ChangeSources +that might be useful with them. All of these ChangeSources are in the +@code{buildbot.changes} module. + +@table @code +@item CVS + +@itemize @bullet +@item freshcvs.FreshCVSSource (connected via TCP to the freshcvs daemon) +@item mail.FCMaildirSource (watching for email sent by a freshcvs daemon) +@item mail.BonsaiMaildirSource (watching for email sent by Bonsai) +@item mail.SyncmailMaildirSource (watching for email sent by syncmail) +@item pb.PBChangeSource (listening for connections from @code{buildbot +sendchange} run in a loginfo script) +@item pb.PBChangeSource (listening for connections from a long-running +@code{contrib/viewcvspoll.py} polling process which examines the ViewCVS +database directly +@end itemize + +@item SVN +@itemize @bullet +@item pb.PBChangeSource (listening for connections from +@code{contrib/svn_buildbot.py} run in a postcommit script) +@item pb.PBChangeSource (listening for connections from a long-running +@code{contrib/svn_watcher.py} or @code{contrib/svnpoller.py} polling +process +@item mail.SVNCommitEmailMaildirSource (watching for email sent by commit-email.pl) +@item svnpoller.SVNPoller (polling the SVN repository) +@end itemize + +@item Darcs +@itemize @bullet +@item pb.PBChangeSource (listening for connections from +@code{contrib/darcs_buildbot.py} in a commit script +@end itemize + +@item Mercurial +@itemize @bullet +@item pb.PBChangeSource (listening for connections from +@code{contrib/hg_buildbot.py} run in an 'incoming' hook) +@item pb.PBChangeSource (listening for connections from +@code{buildbot/changes/hgbuildbot.py} run as an in-process 'changegroup' +hook) +@end itemize + +@item Arch/Bazaar +@itemize @bullet +@item pb.PBChangeSource (listening for connections from +@code{contrib/arch_buildbot.py} run in a commit hook) +@end itemize + +@item Bzr (the newer Bazaar) +@itemize @bullet +@item pb.PBChangeSource (listening for connections from +@code{contrib/bzr_buildbot.py} run in a post-change-branch-tip or commit hook) +@item @code{contrib/bzr_buildbot.py}'s BzrPoller (polling the Bzr repository) +@end itemize + +@item Git +@itemize @bullet +@item pb.PBChangeSource (listening for connections from +@code{contrib/git_buildbot.py} run in the post-receive hook) +@end itemize + +@end table + +All VC systems can be driven by a PBChangeSource and the +@code{buildbot sendchange} tool run from some form of commit script. +If you write an email parsing function, they can also all be driven by +a suitable @code{MaildirSource}. + + +@node Choosing ChangeSources +@section Choosing ChangeSources + +The @code{master.cfg} configuration file has a dictionary key named +@code{BuildmasterConfig['change_source']}, which holds the active +@code{IChangeSource} object. The config file will typically create an +object from one of the classes described below and stuff it into this +key. + +Each buildmaster typically has just a single ChangeSource, since it is +only watching a single source tree. But if, for some reason, you need +multiple sources, just set @code{c['change_source']} to a list of +ChangeSources.. it will accept that too. + +@example +s = FreshCVSSourceNewcred(host="host", port=4519, + user="alice", passwd="secret", + prefix="Twisted") +BuildmasterConfig['change_source'] = [s] +@end example + +Each source tree has a nominal @code{top}. Each Change has a list of +filenames, which are all relative to this top location. The +ChangeSource is responsible for doing whatever is necessary to +accomplish this. Most sources have a @code{prefix} argument: a partial +pathname which is stripped from the front of all filenames provided to +that @code{ChangeSource}. Files which are outside this sub-tree are +ignored by the changesource: it does not generate Changes for those +files. + + +@node CVSToys - PBService +@section CVSToys - PBService + +@csindex buildbot.changes.freshcvs.FreshCVSSource + +The @uref{http://purl.net/net/CVSToys, CVSToys} package provides a +server which runs on the machine that hosts the CVS repository it +watches. It has a variety of ways to distribute commit notifications, +and offers a flexible regexp-based way to filter out uninteresting +changes. One of the notification options is named @code{PBService} and +works by listening on a TCP port for clients. These clients subscribe +to hear about commit notifications. + +The buildmaster has a CVSToys-compatible @code{PBService} client built +in. There are two versions of it, one for old versions of CVSToys +(1.0.9 and earlier) which used the @code{oldcred} authentication +framework, and one for newer versions (1.0.10 and later) which use +@code{newcred}. Both are classes in the +@code{buildbot.changes.freshcvs} package. + +@code{FreshCVSSourceNewcred} objects are created with the following +parameters: + +@table @samp + +@item @code{host} and @code{port} +these specify where the CVSToys server can be reached + +@item @code{user} and @code{passwd} +these specify the login information for the CVSToys server +(@code{freshcvs}). These must match the server's values, which are +defined in the @code{freshCfg} configuration file (which lives in the +CVSROOT directory of the repository). + +@item @code{prefix} +this is the prefix to be found and stripped from filenames delivered +by the CVSToys server. Most projects live in sub-directories of the +main repository, as siblings of the CVSROOT sub-directory, so +typically this prefix is set to that top sub-directory name. + +@end table + +@heading Example + +To set up the freshCVS server, add a statement like the following to +your @file{freshCfg} file: + +@example +pb = ConfigurationSet([ + (None, None, None, PBService(userpass=('foo', 'bar'), port=4519)), + ]) +@end example + +This will announce all changes to a client which connects to port 4519 +using a username of 'foo' and a password of 'bar'. + +Then add a clause like this to your buildmaster's @file{master.cfg}: + +@example +BuildmasterConfig['change_source'] = FreshCVSSource("cvs.example.com", 4519, + "foo", "bar", + prefix="glib/") +@end example + +where "cvs.example.com" is the host that is running the FreshCVS daemon, and +"glib" is the top-level directory (relative to the repository's root) where +all your source code lives. Most projects keep one or more projects in the +same repository (along with CVSROOT/ to hold admin files like loginfo and +freshCfg); the prefix= argument tells the buildmaster to ignore everything +outside that directory, and to strip that common prefix from all pathnames +it handles. + + +@node Mail-parsing ChangeSources +@section Mail-parsing ChangeSources + +Many projects publish information about changes to their source tree +by sending an email message out to a mailing list, frequently named +PROJECT-commits or PROJECT-changes. Each message usually contains a +description of the change (who made the change, which files were +affected) and sometimes a copy of the diff. Humans can subscribe to +this list to stay informed about what's happening to the source tree. + +The Buildbot can also be subscribed to a -commits mailing list, and +can trigger builds in response to Changes that it hears about. The +buildmaster admin needs to arrange for these email messages to arrive +in a place where the buildmaster can find them, and configure the +buildmaster to parse the messages correctly. Once that is in place, +the email parser will create Change objects and deliver them to the +Schedulers (see @pxref{Change Sources and Schedulers}) just +like any other ChangeSource. + +There are two components to setting up an email-based ChangeSource. +The first is to route the email messages to the buildmaster, which is +done by dropping them into a ``maildir''. The second is to actually +parse the messages, which is highly dependent upon the tool that was +used to create them. Each VC system has a collection of favorite +change-emailing tools, and each has a slightly different format, so +each has a different parsing function. There is a separate +ChangeSource variant for each parsing function. + +Once you've chosen a maildir location and a parsing function, create +the change source and put it in @code{c['change_source']}: + +@example +from buildbot.changes.mail import SyncmailMaildirSource +c['change_source'] = SyncmailMaildirSource("~/maildir-buildbot", + prefix="/trunk/") +@end example + +@menu +* Subscribing the Buildmaster:: +* Using Maildirs:: +* Parsing Email Change Messages:: +@end menu + +@node Subscribing the Buildmaster +@subsection Subscribing the Buildmaster + +The recommended way to install the buildbot is to create a dedicated +account for the buildmaster. If you do this, the account will probably +have a distinct email address (perhaps +@email{buildmaster@@example.org}). Then just arrange for this +account's email to be delivered to a suitable maildir (described in +the next section). + +If the buildbot does not have its own account, ``extension addresses'' +can be used to distinguish between email intended for the buildmaster +and email intended for the rest of the account. In most modern MTAs, +the e.g. @code{foo@@example.org} account has control over every email +address at example.org which begins with "foo", such that email +addressed to @email{account-foo@@example.org} can be delivered to a +different destination than @email{account-bar@@example.org}. qmail +does this by using separate .qmail files for the two destinations +(@file{.qmail-foo} and @file{.qmail-bar}, with @file{.qmail} +controlling the base address and @file{.qmail-default} controlling all +other extensions). Other MTAs have similar mechanisms. + +Thus you can assign an extension address like +@email{foo-buildmaster@@example.org} to the buildmaster, and retain +@email{foo@@example.org} for your own use. + + +@node Using Maildirs +@subsection Using Maildirs + +A ``maildir'' is a simple directory structure originally developed for +qmail that allows safe atomic update without locking. Create a base +directory with three subdirectories: ``new'', ``tmp'', and ``cur''. +When messages arrive, they are put into a uniquely-named file (using +pids, timestamps, and random numbers) in ``tmp''. When the file is +complete, it is atomically renamed into ``new''. Eventually the +buildmaster notices the file in ``new'', reads and parses the +contents, then moves it into ``cur''. A cronjob can be used to delete +files in ``cur'' at leisure. + +Maildirs are frequently created with the @command{maildirmake} tool, +but a simple @command{mkdir -p ~/MAILDIR/@{cur,new,tmp@}} is pretty much +equivalent. + +Many modern MTAs can deliver directly to maildirs. The usual .forward +or .procmailrc syntax is to name the base directory with a trailing +slash, so something like @code{~/MAILDIR/} . qmail and postfix are +maildir-capable MTAs, and procmail is a maildir-capable MDA (Mail +Delivery Agent). + +For MTAs which cannot put files into maildirs directly, the +``safecat'' tool can be executed from a .forward file to accomplish +the same thing. + +The Buildmaster uses the linux DNotify facility to receive immediate +notification when the maildir's ``new'' directory has changed. When +this facility is not available, it polls the directory for new +messages, every 10 seconds by default. + +@node Parsing Email Change Messages +@subsection Parsing Email Change Messages + +The second component to setting up an email-based ChangeSource is to +parse the actual notices. This is highly dependent upon the VC system +and commit script in use. + +A couple of common tools used to create these change emails are: + +@table @samp + +@item CVS +@table @samp +@item CVSToys MailNotifier +@ref{FCMaildirSource} +@item Bonsai notification +@ref{BonsaiMaildirSource} +@item syncmail +@ref{SyncmailMaildirSource} +@end table + +@item SVN +@table @samp +@item svnmailer +http://opensource.perlig.de/en/svnmailer/ +@item commit-email.pl +@ref{SVNCommitEmailMaildirSource} +@end table + +@item Bzr +@table @samp +@item Launchpad +@ref{BzrLaunchpadEmailMaildirSource} +@end table + +@item Mercurial +@table @samp +@item NotifyExtension +http://www.selenic.com/mercurial/wiki/index.cgi/NotifyExtension +@end table + +@item Git +@table @samp +@item post-receive-email +http://git.kernel.org/?p=git/git.git;a=blob;f=contrib/hooks/post-receive-email;hb=HEAD +@end table + +@end table + + +The following sections describe the parsers available for each of +these tools. + +Most of these parsers accept a @code{prefix=} argument, which is used +to limit the set of files that the buildmaster pays attention to. This +is most useful for systems like CVS and SVN which put multiple +projects in a single repository (or use repository names to indicate +branches). Each filename that appears in the email is tested against +the prefix: if the filename does not start with the prefix, the file +is ignored. If the filename @emph{does} start with the prefix, that +prefix is stripped from the filename before any further processing is +done. Thus the prefix usually ends with a slash. + +@menu +* FCMaildirSource:: +* SyncmailMaildirSource:: +* BonsaiMaildirSource:: +* SVNCommitEmailMaildirSource:: +* BzrLaunchpadEmailMaildirSource:: +@end menu + +@node FCMaildirSource +@subsubsection FCMaildirSource + + +@csindex buildbot.changes.mail.FCMaildirSource + +http://twistedmatrix.com/users/acapnotic/wares/code/CVSToys/ + +This parser works with the CVSToys @code{MailNotification} action, +which will send email to a list of recipients for each commit. This +tends to work better than using @code{/bin/mail} from within the +CVSROOT/loginfo file directly, as CVSToys will batch together all +files changed during the same CVS invocation, and can provide more +information (like creating a ViewCVS URL for each file changed). + +The Buildbot's @code{FCMaildirSource} knows for to parse these CVSToys +messages and turn them into Change objects. It can be given two +parameters: the directory name of the maildir root, and the prefix to +strip. + +@example +from buildbot.changes.mail import FCMaildirSource +c['change_source'] = FCMaildirSource("~/maildir-buildbot") +@end example + +@node SyncmailMaildirSource +@subsubsection SyncmailMaildirSource + +@csindex buildbot.changes.mail.SyncmailMaildirSource + +http://sourceforge.net/projects/cvs-syncmail + +@code{SyncmailMaildirSource} knows how to parse the message format used by +the CVS ``syncmail'' script. + +@example +from buildbot.changes.mail import SyncmailMaildirSource +c['change_source'] = SyncmailMaildirSource("~/maildir-buildbot") +@end example + +@node BonsaiMaildirSource +@subsubsection BonsaiMaildirSource + +@csindex buildbot.changes.mail.BonsaiMaildirSource + +http://www.mozilla.org/bonsai.html + +@code{BonsaiMaildirSource} parses messages sent out by Bonsai, the CVS +tree-management system built by Mozilla. + +@example +from buildbot.changes.mail import BonsaiMaildirSource +c['change_source'] = BonsaiMaildirSource("~/maildir-buildbot") +@end example + +@node SVNCommitEmailMaildirSource +@subsubsection SVNCommitEmailMaildirSource + +@csindex buildbot.changes.mail.SVNCommitEmailMaildirSource + +@code{SVNCommitEmailMaildirSource} parses message sent out by the +@code{commit-email.pl} script, which is included in the Subversion +distribution. + +It does not currently handle branches: all of the Change objects that +it creates will be associated with the default (i.e. trunk) branch. + +@example +from buildbot.changes.mail import SVNCommitEmailMaildirSource +c['change_source'] = SVNCommitEmailMaildirSource("~/maildir-buildbot") +@end example + +@node BzrLaunchpadEmailMaildirSource +@subsubsection BzrLaunchpadEmailMaildirSource + +@csindex buildbot.changes.mail.BzrLaunchpadEmailMaildirSource + +@code{BzrLaunchpadEmailMaildirSource} parses the mails that are sent to +addresses that subscribe to branch revision notifications for a bzr branch +hosted on Launchpad. + +The branch name defaults to @code{lp:}. For example +@code{lp:~maria-captains/maria/5.1}. + +If only a single branch is used, the default branch name can be changed by +setting @code{defaultBranch}. + +For multiple branches, pass a dictionary as the value of the @code{branchMap} +option to map specific repository paths to specific branch names (see example +below). The leading @code{lp:} prefix of the path is optional. + +The @code{prefix} option is not supported (it is silently ignored). Use the +@code{branchMap} and @code{defaultBranch} instead to assign changes to +branches (and just do not subscribe the buildbot to branches that are not of +interest). + +The revision number is obtained from the email text. The bzr revision id is +not available in the mails sent by Launchpad. However, it is possible to set +the bzr @code{append_revisions_only} option for public shared repositories to +avoid new pushes of merges changing the meaning of old revision numbers. + +@verbatim +from buildbot.changes.mail import BzrLaunchpadEmailMaildirSource +bm = { 'lp:~maria-captains/maria/5.1' : '5.1', 'lp:~maria-captains/maria/6.0' : '6.0' } +c['change_source'] = BzrLaunchpadEmailMaildirSource("~/maildir-buildbot", branchMap = bm) +@end verbatim + + +@node PBChangeSource +@section PBChangeSource + +@csindex buildbot.changes.pb.PBChangeSource + +The last kind of ChangeSource actually listens on a TCP port for +clients to connect and push change notices @emph{into} the +Buildmaster. This is used by the built-in @code{buildbot sendchange} +notification tool, as well as the VC-specific +@file{contrib/svn_buildbot.py}, @file{contrib/arch_buildbot.py}, +@file{contrib/hg_buildbot.py} tools, and the +@code{buildbot.changes.hgbuildbot} hook. These tools are run by the +repository (in a commit hook script), and connect to the buildmaster +directly each time a file is comitted. This is also useful for +creating new kinds of change sources that work on a @code{push} model +instead of some kind of subscription scheme, for example a script +which is run out of an email .forward file. + +This ChangeSource can be configured to listen on its own TCP port, or +it can share the port that the buildmaster is already using for the +buildslaves to connect. (This is possible because the +@code{PBChangeSource} uses the same protocol as the buildslaves, and +they can be distinguished by the @code{username} attribute used when +the initial connection is established). It might be useful to have it +listen on a different port if, for example, you wanted to establish +different firewall rules for that port. You could allow only the SVN +repository machine access to the @code{PBChangeSource} port, while +allowing only the buildslave machines access to the slave port. Or you +could just expose one port and run everything over it. @emph{Note: +this feature is not yet implemented, the PBChangeSource will always +share the slave port and will always have a @code{user} name of +@code{change}, and a passwd of @code{changepw}. These limitations will +be removed in the future.}. + + +The @code{PBChangeSource} is created with the following arguments. All +are optional. + +@table @samp +@item @code{port} +which port to listen on. If @code{None} (which is the default), it +shares the port used for buildslave connections. @emph{Not +Implemented, always set to @code{None}}. + +@item @code{user} and @code{passwd} +The user/passwd account information that the client program must use +to connect. Defaults to @code{change} and @code{changepw}. @emph{Not +Implemented, @code{user} is currently always set to @code{change}, +@code{passwd} is always set to @code{changepw}}. + +@item @code{prefix} +The prefix to be found and stripped from filenames delivered over the +connection. Any filenames which do not start with this prefix will be +removed. If all the filenames in a given Change are removed, the that +whole Change will be dropped. This string should probably end with a +directory separator. + +This is useful for changes coming from version control systems that +represent branches as parent directories within the repository (like +SVN and Perforce). Use a prefix of 'trunk/' or +'project/branches/foobranch/' to only follow one branch and to get +correct tree-relative filenames. Without a prefix, the PBChangeSource +will probably deliver Changes with filenames like @file{trunk/foo.c} +instead of just @file{foo.c}. Of course this also depends upon the +tool sending the Changes in (like @command{buildbot sendchange}) and +what filenames it is delivering: that tool may be filtering and +stripping prefixes at the sending end. + +@end table + +@node P4Source +@section P4Source + +@csindex buildbot.changes.p4poller.P4Source + +The @code{P4Source} periodically polls a @uref{http://www.perforce.com/, +Perforce} depot for changes. It accepts the following arguments: + +@table @samp +@item @code{p4base} +The base depot path to watch, without the trailing '/...'. + +@item @code{p4port} +The Perforce server to connect to (as host:port). + +@item @code{p4user} +The Perforce user. + +@item @code{p4passwd} +The Perforce password. + +@item @code{p4bin} +An optional string parameter. Specify the location of the perforce command +line binary (p4). You only need to do this if the perforce binary is not +in the path of the buildbot user. Defaults to ``p4''. + +@item @code{split_file} +A function that maps a pathname, without the leading @code{p4base}, to a +(branch, filename) tuple. The default just returns (None, branchfile), +which effectively disables branch support. You should supply a function +which understands your repository structure. + +@item @code{pollinterval} +How often to poll, in seconds. Defaults to 600 (10 minutes). + +@item @code{histmax} +The maximum number of changes to inspect at a time. If more than this +number occur since the last poll, older changes will be silently +ignored. +@end table + +@heading Example + +This configuration uses the @code{P4PORT}, @code{P4USER}, and @code{P4PASSWD} +specified in the buildmaster's environment. It watches a project in which the +branch name is simply the next path component, and the file is all path +components after. + +@example +from buildbot.changes import p4poller +s = p4poller.P4Source(p4base='//depot/project/', + split_file=lambda branchfile: branchfile.split('/',1), + ) +c['change_source'] = s +@end example + +@node BonsaiPoller +@section BonsaiPoller + +@csindex buildbot.changes.bonsaipoller.BonsaiPoller + +The @code{BonsaiPoller} periodically polls a Bonsai server. This is a +CGI script accessed through a web server that provides information +about a CVS tree, for example the Mozilla bonsai server at +@uref{http://bonsai.mozilla.org}. Bonsai servers are usable by both +humans and machines. In this case, the buildbot's change source forms +a query which asks about any files in the specified branch which have +changed since the last query. + +Please take a look at the BonsaiPoller docstring for details about the +arguments it accepts. + + +@node SVNPoller +@section SVNPoller + +@csindex buildbot.changes.svnpoller.SVNPoller + +The @code{buildbot.changes.svnpoller.SVNPoller} is a ChangeSource +which periodically polls a @uref{http://subversion.tigris.org/, +Subversion} repository for new revisions, by running the @code{svn +log} command in a subshell. It can watch a single branch or multiple +branches. + +@code{SVNPoller} accepts the following arguments: + +@table @code +@item svnurl +The base URL path to watch, like +@code{svn://svn.twistedmatrix.com/svn/Twisted/trunk}, or +@code{http://divmod.org/svn/Divmod/}, or even +@code{file:///home/svn/Repository/ProjectA/branches/1.5/}. This must +include the access scheme, the location of the repository (both the +hostname for remote ones, and any additional directory names necessary +to get to the repository), and the sub-path within the repository's +virtual filesystem for the project and branch of interest. + +The @code{SVNPoller} will only pay attention to files inside the +subdirectory specified by the complete svnurl. + +@item split_file +A function to convert pathnames into (branch, relative_pathname) +tuples. Use this to explain your repository's branch-naming policy to +@code{SVNPoller}. This function must accept a single string and return +a two-entry tuple. There are a few utility functions in +@code{buildbot.changes.svnpoller} that can be used as a +@code{split_file} function, see below for details. + +The default value always returns (None, path), which indicates that +all files are on the trunk. + +Subclasses of @code{SVNPoller} can override the @code{split_file} +method instead of using the @code{split_file=} argument. + +@item svnuser +An optional string parameter. If set, the @code{--user} argument will +be added to all @code{svn} commands. Use this if you have to +authenticate to the svn server before you can do @code{svn info} or +@code{svn log} commands. + +@item svnpasswd +Like @code{svnuser}, this will cause a @code{--password} argument to +be passed to all svn commands. + +@item pollinterval +How often to poll, in seconds. Defaults to 600 (checking once every 10 +minutes). Lower this if you want the buildbot to notice changes +faster, raise it if you want to reduce the network and CPU load on +your svn server. Please be considerate of public SVN repositories by +using a large interval when polling them. + +@item histmax +The maximum number of changes to inspect at a time. Every POLLINTERVAL +seconds, the @code{SVNPoller} asks for the last HISTMAX changes and +looks through them for any ones it does not already know about. If +more than HISTMAX revisions have been committed since the last poll, +older changes will be silently ignored. Larger values of histmax will +cause more time and memory to be consumed on each poll attempt. +@code{histmax} defaults to 100. + +@item svnbin +This controls the @code{svn} executable to use. If subversion is +installed in a weird place on your system (outside of the +buildmaster's @code{$PATH}), use this to tell @code{SVNPoller} where +to find it. The default value of ``svn'' will almost always be +sufficient. + +@item revlinktmpl +This parameter allows a link to be provided for each revision (for example, +to websvn or viewvc). These links appear anywhere changes are shown, such +as on build or change pages. The proper form for this parameter is an URL +with the portion that will substitute for a revision number replaced by +''%s''. For example, @code{'http://myserver/websvn/revision.php?rev=%s'} +could be used to cause revision links to be created to a websvn repository +viewer. + +@end table + +@heading Branches + +Each source file that is tracked by a Subversion repository has a +fully-qualified SVN URL in the following form: +(REPOURL)(PROJECT-plus-BRANCH)(FILEPATH). When you create the +@code{SVNPoller}, you give it a @code{svnurl} value that includes all +of the REPOURL and possibly some portion of the PROJECT-plus-BRANCH +string. The @code{SVNPoller} is responsible for producing Changes that +contain a branch name and a FILEPATH (which is relative to the top of +a checked-out tree). The details of how these strings are split up +depend upon how your repository names its branches. + +@subheading PROJECT/BRANCHNAME/FILEPATH repositories + +One common layout is to have all the various projects that share a +repository get a single top-level directory each. Then under a given +project's directory, you get two subdirectories, one named ``trunk'' +and another named ``branches''. Under ``branches'' you have a bunch of +other directories, one per branch, with names like ``1.5.x'' and +``testing''. It is also common to see directories like ``tags'' and +``releases'' next to ``branches'' and ``trunk''. + +For example, the Twisted project has a subversion server on +``svn.twistedmatrix.com'' that hosts several sub-projects. The +repository is available through a SCHEME of ``svn:''. The primary +sub-project is Twisted, of course, with a repository root of +``svn://svn.twistedmatrix.com/svn/Twisted''. Another sub-project is +Informant, with a root of +``svn://svn.twistedmatrix.com/svn/Informant'', etc. Inside any +checked-out Twisted tree, there is a file named bin/trial (which is +used to run unit test suites). + +The trunk for Twisted is in +``svn://svn.twistedmatrix.com/svn/Twisted/trunk'', and the +fully-qualified SVN URL for the trunk version of @code{trial} would be +``svn://svn.twistedmatrix.com/svn/Twisted/trunk/bin/trial''. The same +SVNURL for that file on a branch named ``1.5.x'' would be +``svn://svn.twistedmatrix.com/svn/Twisted/branches/1.5.x/bin/trial''. + +To set up a @code{SVNPoller} that watches the Twisted trunk (and +nothing else), we would use the following: + +@example +from buildbot.changes.svnpoller import SVNPoller +c['change_source'] = SVNPoller("svn://svn.twistedmatrix.com/svn/Twisted/trunk") +@end example + +In this case, every Change that our @code{SVNPoller} produces will +have @code{.branch=None}, to indicate that the Change is on the trunk. +No other sub-projects or branches will be tracked. + +If we want our ChangeSource to follow multiple branches, we have to do +two things. First we have to change our @code{svnurl=} argument to +watch more than just ``.../Twisted/trunk''. We will set it to +``.../Twisted'' so that we'll see both the trunk and all the branches. +Second, we have to tell @code{SVNPoller} how to split the +(PROJECT-plus-BRANCH)(FILEPATH) strings it gets from the repository +out into (BRANCH) and (FILEPATH) pairs. + +We do the latter by providing a ``split_file'' function. This function +is responsible for splitting something like +``branches/1.5.x/bin/trial'' into @code{branch}=''branches/1.5.x'' and +@code{filepath}=''bin/trial''. This function is always given a string +that names a file relative to the subdirectory pointed to by the +@code{SVNPoller}'s @code{svnurl=} argument. It is expected to return a +(BRANCHNAME, FILEPATH) tuple (in which FILEPATH is relative to the +branch indicated), or None to indicate that the file is outside any +project of interest. + +(note that we want to see ``branches/1.5.x'' rather than just +``1.5.x'' because when we perform the SVN checkout, we will probably +append the branch name to the baseURL, which requires that we keep the +``branches'' component in there. Other VC schemes use a different +approach towards branches and may not require this artifact.) + +If your repository uses this same PROJECT/BRANCH/FILEPATH naming +scheme, the following function will work: + +@example +def split_file_branches(path): + pieces = path.split('/') + if pieces[0] == 'trunk': + return (None, '/'.join(pieces[1:])) + elif pieces[0] == 'branches': + return ('/'.join(pieces[0:2]), + '/'.join(pieces[2:])) + else: + return None +@end example + +This function is provided as +@code{buildbot.changes.svnpoller.split_file_branches} for your +convenience. So to have our Twisted-watching @code{SVNPoller} follow +multiple branches, we would use this: + +@example +from buildbot.changes.svnpoller import SVNPoller, split_file_branches +c['change_source'] = SVNPoller("svn://svn.twistedmatrix.com/svn/Twisted", + split_file=split_file_branches) +@end example + +Changes for all sorts of branches (with names like ``branches/1.5.x'', +and None to indicate the trunk) will be delivered to the Schedulers. +Each Scheduler is then free to use or ignore each branch as it sees +fit. + +@subheading BRANCHNAME/PROJECT/FILEPATH repositories + +Another common way to organize a Subversion repository is to put the +branch name at the top, and the projects underneath. This is +especially frequent when there are a number of related sub-projects +that all get released in a group. + +For example, Divmod.org hosts a project named ``Nevow'' as well as one +named ``Quotient''. In a checked-out Nevow tree there is a directory +named ``formless'' that contains a python source file named +``webform.py''. This repository is accessible via webdav (and thus +uses an ``http:'' scheme) through the divmod.org hostname. There are +many branches in this repository, and they use a +(BRANCHNAME)/(PROJECT) naming policy. + +The fully-qualified SVN URL for the trunk version of webform.py is +@code{http://divmod.org/svn/Divmod/trunk/Nevow/formless/webform.py}. +You can do an @code{svn co} with that URL and get a copy of the latest +version. The 1.5.x branch version of this file would have a URL of +@code{http://divmod.org/svn/Divmod/branches/1.5.x/Nevow/formless/webform.py}. +The whole Nevow trunk would be checked out with +@code{http://divmod.org/svn/Divmod/trunk/Nevow}, while the Quotient +trunk would be checked out using +@code{http://divmod.org/svn/Divmod/trunk/Quotient}. + +Now suppose we want to have an @code{SVNPoller} that only cares about +the Nevow trunk. This case looks just like the PROJECT/BRANCH layout +described earlier: + +@example +from buildbot.changes.svnpoller import SVNPoller +c['change_source'] = SVNPoller("http://divmod.org/svn/Divmod/trunk/Nevow") +@end example + +But what happens when we want to track multiple Nevow branches? We +have to point our @code{svnurl=} high enough to see all those +branches, but we also don't want to include Quotient changes (since +we're only building Nevow). To accomplish this, we must rely upon the +@code{split_file} function to help us tell the difference between +files that belong to Nevow and those that belong to Quotient, as well +as figuring out which branch each one is on. + +@example +from buildbot.changes.svnpoller import SVNPoller +c['change_source'] = SVNPoller("http://divmod.org/svn/Divmod", + split_file=my_file_splitter) +@end example + +The @code{my_file_splitter} function will be called with +repository-relative pathnames like: + +@table @code +@item trunk/Nevow/formless/webform.py +This is a Nevow file, on the trunk. We want the Change that includes this +to see a filename of @code{formless/webform.py"}, and a branch of None + +@item branches/1.5.x/Nevow/formless/webform.py +This is a Nevow file, on a branch. We want to get +branch=''branches/1.5.x'' and filename=''formless/webform.py''. + +@item trunk/Quotient/setup.py +This is a Quotient file, so we want to ignore it by having +@code{my_file_splitter} return None. + +@item branches/1.5.x/Quotient/setup.py +This is also a Quotient file, which should be ignored. +@end table + +The following definition for @code{my_file_splitter} will do the job: + +@example +def my_file_splitter(path): + pieces = path.split('/') + if pieces[0] == 'trunk': + branch = None + pieces.pop(0) # remove 'trunk' + elif pieces[0] == 'branches': + pieces.pop(0) # remove 'branches' + # grab branch name + branch = 'branches/' + pieces.pop(0) + else: + return None # something weird + projectname = pieces.pop(0) + if projectname != 'Nevow': + return None # wrong project + return (branch, '/'.join(pieces)) +@end example + +@node MercurialHook +@section MercurialHook + +Since Mercurial is written in python, the hook script can invoke +Buildbot's @code{sendchange} function directly, rather than having to +spawn an external process. This function delivers the same sort of +changes as @code{buildbot sendchange} and the various hook scripts in +contrib/, so you'll need to add a @code{pb.PBChangeSource} to your +buildmaster to receive these changes. + +To set this up, first choose a Mercurial repository that represents +your central ``official'' source tree. This will be the same +repository that your buildslaves will eventually pull from. Install +Buildbot on the machine that hosts this repository, using the same +version of python as Mercurial is using (so that the Mercurial hook +can import code from buildbot). Then add the following to the +@code{.hg/hgrc} file in that repository, replacing the buildmaster +hostname/portnumber as appropriate for your buildbot: + +@example +[hooks] +changegroup.buildbot = python:buildbot.changes.hgbuildbot.hook + +[hgbuildbot] +master = buildmaster.example.org:9987 +@end example + +(Note that Mercurial lets you define multiple @code{changegroup} hooks +by giving them distinct names, like @code{changegroup.foo} and +@code{changegroup.bar}, which is why we use +@code{changegroup.buildbot} in this example. There is nothing magical +about the ``buildbot'' suffix in the hook name. The +@code{[hgbuildbot]} section @emph{is} special, however, as it is the +only section that the buildbot hook pays attention to.) + +Also note that this runs as a @code{changegroup} hook, rather than as +an @code{incoming} hook. The @code{changegroup} hook is run with +multiple revisions at a time (say, if multiple revisions are being +pushed to this repository in a single @command{hg push} command), +whereas the @code{incoming} hook is run with just one revision at a +time. The @code{hgbuildbot.hook} function will only work with the +@code{changegroup} hook. + +The @code{[hgbuildbot]} section has two other parameters that you +might specify, both of which control the name of the branch that is +attached to the changes coming from this hook. + +One common branch naming policy for Mercurial repositories is to use +it just like Darcs: each branch goes into a separate repository, and +all the branches for a single project share a common parent directory. +For example, you might have @file{/var/repos/PROJECT/trunk/} and +@file{/var/repos/PROJECT/release}. To use this style, use the +@code{branchtype = dirname} setting, which simply uses the last +component of the repository's enclosing directory as the branch name: + +@example +[hgbuildbot] +master = buildmaster.example.org:9987 +branchtype = dirname +@end example + +Another approach is to use Mercurial's built-in branches (the kind +created with @command{hg branch} and listed with @command{hg +branches}). This feature associates persistent names with particular +lines of descent within a single repository. (note that the buildbot +@code{source.Mercurial} checkout step does not yet support this kind +of branch). To have the commit hook deliver this sort of branch name +with the Change object, use @code{branchtype = inrepo}: + +@example +[hgbuildbot] +master = buildmaster.example.org:9987 +branchtype = inrepo +@end example + +Finally, if you want to simply specify the branchname directly, for +all changes, use @code{branch = BRANCHNAME}. This overrides +@code{branchtype}: + +@example +[hgbuildbot] +master = buildmaster.example.org:9987 +branch = trunk +@end example + +If you use @code{branch=} like this, you'll need to put a separate +.hgrc in each repository. If you use @code{branchtype=}, you may be +able to use the same .hgrc for all your repositories, stored in +@file{~/.hgrc} or @file{/etc/mercurial/hgrc}. + + +@node Bzr Hook +@section Bzr Hook + +Bzr is also written in Python, and the Bzr hook depends on Twisted to send the +changes. + +To install, put @code{contrib/bzr_buildbot.py} in one of your plugins +locations a bzr plugins directory (e.g., +@code{~/.bazaar/plugins}). Then, in one of your bazaar conf files (e.g., +@code{~/.bazaar/locations.conf}), set the location you want to connect with buildbot +with these keys: + +@table @code +@item buildbot_on +one of 'commit', 'push, or 'change'. Turns the plugin on to report changes via +commit, changes via push, or any changes to the trunk. 'change' is +recommended. + +@item buildbot_server +(required to send to a buildbot master) the URL of the buildbot master to +which you will connect (as of this writing, the same server and port to which +slaves connect). + +@item buildbot_port +(optional, defaults to 9989) the port of the buildbot master to which you will +connect (as of this writing, the same server and port to which slaves connect) + +@item buildbot_pqm +(optional, defaults to not pqm) Normally, the user that commits the revision +is the user that is responsible for the change. When run in a pqm (Patch Queue +Manager, see https://launchpad.net/pqm) environment, the user that commits is +the Patch Queue Manager, and the user that committed the *parent* revision is +responsible for the change. To turn on the pqm mode, set this value to any of +(case-insensitive) "Yes", "Y", "True", or "T". + +@item buildbot_dry_run +(optional, defaults to not a dry run) Normally, the post-commit hook will +attempt to communicate with the configured buildbot server and port. If this +parameter is included and any of (case-insensitive) "Yes", "Y", "True", or +"T", then the hook will simply print what it would have sent, but not attempt +to contact the buildbot master. + +@item buildbot_send_branch_name +(optional, defaults to not sending the branch name) If your buildbot's bzr +source build step uses a repourl, do *not* turn this on. If your buildbot's +bzr build step uses a baseURL, then you may set this value to any of +(case-insensitive) "Yes", "Y", "True", or "T" to have the buildbot master +append the branch name to the baseURL. + +@end table + +When buildbot no longer has a hardcoded password, it will be a configuration +option here as well. + +Here's a simple example that you might have in your +@code{~/.bazaar/locations.conf}. + +@example +[chroot-*:///var/local/myrepo/mybranch] +buildbot_on = change +buildbot_server = localhost +@end example + +@node Bzr Poller +@section Bzr Poller + +If you cannot insert a Bzr hook in the server, you can use the Bzr Poller. To +use, put @code{contrib/bzr_buildbot.py} somewhere that your buildbot +configuration can import it. Even putting it in the same directory as the master.cfg +should work. Install the poller in the buildbot configuration as with any +other change source. Minimally, provide a URL that you want to poll (bzr://, +bzr+ssh://, or lp:), though make sure the buildbot user has necessary +privileges. You may also want to specify these optional values. + +@table @code +@item poll_interval +The number of seconds to wait between polls. Defaults to 10 minutes. + +@item branch_name +Any value to be used as the branch name. Defaults to None, or specify a +string, or specify the constants from @code{bzr_buildbot.py} SHORT or FULL to +get the short branch name or full branch address. + +@item blame_merge_author +normally, the user that commits the revision is the user that is responsible +for the change. When run in a pqm (Patch Queue Manager, see +https://launchpad.net/pqm) environment, the user that commits is the Patch +Queue Manager, and the user that committed the merged, *parent* revision is +responsible for the change. set this value to True if this is pointed against +a PQM-managed branch. +@end table + + diff --git a/docs/cmdline.texinfo b/docs/cmdline.texinfo new file mode 100644 index 00000000000..889860cd28b --- /dev/null +++ b/docs/cmdline.texinfo @@ -0,0 +1,776 @@ +The @command{buildbot} command-line tool can be used to start or stop a +buildmaster or buildbot, and to interact with a running buildmaster. +Some of its subcommands are intended for buildmaster admins, while +some are for developers who are editing the code that the buildbot is +monitoring. + +@menu +* Administrator Tools:: +* Developer Tools:: +* Other Tools:: +* .buildbot config directory:: +@end menu + +@node Administrator Tools +@section Administrator Tools + +The following @command{buildbot} sub-commands are intended for +buildmaster administrators: + +@heading create-master + +This creates a new directory and populates it with files that allow it +to be used as a buildmaster's base directory. + +@example +buildbot create-master BASEDIR +@end example + +@heading create-slave + +This creates a new directory and populates it with files that let it +be used as a buildslave's base directory. You must provide several +arguments, which are used to create the initial @file{buildbot.tac} +file. + +@example +buildbot create-slave @var{BASEDIR} @var{MASTERHOST}:@var{PORT} @var{SLAVENAME} @var{PASSWORD} +@end example + +@heading start + +This starts a buildmaster or buildslave which was already created in +the given base directory. The daemon is launched in the background, +with events logged to a file named @file{twistd.log}. + +@example +buildbot start BASEDIR +@end example + +@heading stop + +This terminates the daemon (either buildmaster or buildslave) running +in the given directory. + +@example +buildbot stop BASEDIR +@end example + +@heading sighup + +This sends a SIGHUP to the buildmaster running in the given directory, +which causes it to re-read its @file{master.cfg} file. + +@example +buildbot sighup BASEDIR +@end example + +@node Developer Tools +@section Developer Tools + +These tools are provided for use by the developers who are working on +the code that the buildbot is monitoring. + +@menu +* statuslog:: +* statusgui:: +* try:: +@end menu + +@node statuslog +@subsection statuslog + +@example +buildbot statuslog --master @var{MASTERHOST}:@var{PORT} +@end example + +This command starts a simple text-based status client, one which just +prints out a new line each time an event occurs on the buildmaster. + +The @option{--master} option provides the location of the +@code{buildbot.status.client.PBListener} status port, used to deliver +build information to realtime status clients. The option is always in +the form of a string, with hostname and port number separated by a +colon (@code{HOSTNAME:PORTNUM}). Note that this port is @emph{not} the +same as the slaveport (although a future version may allow the same +port number to be used for both purposes). If you get an error message +to the effect of ``Failure: twisted.cred.error.UnauthorizedLogin:'', +this may indicate that you are connecting to the slaveport rather than +a @code{PBListener} port. + +The @option{--master} option can also be provided by the +@code{masterstatus} name in @file{.buildbot/options} (@pxref{.buildbot +config directory}). + +@node statusgui +@subsection statusgui + +@cindex statusgui + +If you have set up a PBListener (@pxref{PBListener}), you will be able +to monitor your Buildbot using a simple Gtk+ application invoked with +the @code{buildbot statusgui} command: + +@example +buildbot statusgui --master @var{MASTERHOST}:@var{PORT} +@end example + +This command starts a simple Gtk+-based status client, which contains a few +boxes for each Builder that change color as events occur. It uses the same +@option{--master} argument and @code{masterstatus} option as the +@command{buildbot statuslog} command (@pxref{statuslog}). + +@node try +@subsection try + +This lets a developer to ask the question ``What would happen if I +committed this patch right now?''. It runs the unit test suite (across +multiple build platforms) on the developer's current code, allowing +them to make sure they will not break the tree when they finally +commit their changes. + +The @command{buildbot try} command is meant to be run from within a +developer's local tree, and starts by figuring out the base revision +of that tree (what revision was current the last time the tree was +updated), and a patch that can be applied to that revision of the tree +to make it match the developer's copy. This (revision, patch) pair is +then sent to the buildmaster, which runs a build with that +SourceStamp. If you want, the tool will emit status messages as the +builds run, and will not terminate until the first failure has been +detected (or the last success). + +There is an alternate form which accepts a pre-made patch file +(typically the output of a command like 'svn diff'). This ``--diff'' +form does not require a local tree to run from. See @xref{try --diff}. + +For this command to work, several pieces must be in place: + + +@heading TryScheduler + +@slindex buildbot.scheduler.Try_Jobdir +@slindex buildbot.scheduler.Try_Userpass + +The buildmaster must have a @code{scheduler.Try} instance in +the config file's @code{c['schedulers']} list. This lets the +administrator control who may initiate these ``trial'' builds, which +branches are eligible for trial builds, and which Builders should be +used for them. + +The @code{TryScheduler} has various means to accept build requests: +all of them enforce more security than the usual buildmaster ports do. +Any source code being built can be used to compromise the buildslave +accounts, but in general that code must be checked out from the VC +repository first, so only people with commit privileges can get +control of the buildslaves. The usual force-build control channels can +waste buildslave time but do not allow arbitrary commands to be +executed by people who don't have those commit privileges. However, +the source code patch that is provided with the trial build does not +have to go through the VC system first, so it is important to make +sure these builds cannot be abused by a non-committer to acquire as +much control over the buildslaves as a committer has. Ideally, only +developers who have commit access to the VC repository would be able +to start trial builds, but unfortunately the buildmaster does not, in +general, have access to VC system's user list. + +As a result, the @code{TryScheduler} requires a bit more +configuration. There are currently two ways to set this up: + +@table @strong +@item jobdir (ssh) + +This approach creates a command queue directory, called the +``jobdir'', in the buildmaster's working directory. The buildmaster +admin sets the ownership and permissions of this directory to only +grant write access to the desired set of developers, all of whom must +have accounts on the machine. The @code{buildbot try} command creates +a special file containing the source stamp information and drops it in +the jobdir, just like a standard maildir. When the buildmaster notices +the new file, it unpacks the information inside and starts the builds. + +The config file entries used by 'buildbot try' either specify a local +queuedir (for which write and mv are used) or a remote one (using scp +and ssh). + +The advantage of this scheme is that it is quite secure, the +disadvantage is that it requires fiddling outside the buildmaster +config (to set the permissions on the jobdir correctly). If the +buildmaster machine happens to also house the VC repository, then it +can be fairly easy to keep the VC userlist in sync with the +trial-build userlist. If they are on different machines, this will be +much more of a hassle. It may also involve granting developer accounts +on a machine that would not otherwise require them. + +To implement this, the buildslave invokes 'ssh -l username host +buildbot tryserver ARGS', passing the patch contents over stdin. The +arguments must include the inlet directory and the revision +information. + +@item user+password (PB) + +In this approach, each developer gets a username/password pair, which +are all listed in the buildmaster's configuration file. When the +developer runs @code{buildbot try}, their machine connects to the +buildmaster via PB and authenticates themselves using that username +and password, then sends a PB command to start the trial build. + +The advantage of this scheme is that the entire configuration is +performed inside the buildmaster's config file. The disadvantages are +that it is less secure (while the ``cred'' authentication system does +not expose the password in plaintext over the wire, it does not offer +most of the other security properties that SSH does). In addition, the +buildmaster admin is responsible for maintaining the username/password +list, adding and deleting entries as developers come and go. + +@end table + + +For example, to set up the ``jobdir'' style of trial build, using a +command queue directory of @file{MASTERDIR/jobdir} (and assuming that +all your project developers were members of the @code{developers} unix +group), you would first create that directory (with @command{mkdir +MASTERDIR/jobdir MASTERDIR/jobdir/new MASTERDIR/jobdir/cur +MASTERDIR/jobdir/tmp; chgrp developers MASTERDIR/jobdir +MASTERDIR/jobdir/*; chmod g+rwx,o-rwx MASTERDIR/jobdir +MASTERDIR/jobdir/*}), and then use the following scheduler in the +buildmaster's config file: + +@example +from buildbot.scheduler import Try_Jobdir +s = Try_Jobdir("try1", ["full-linux", "full-netbsd", "full-OSX"], + jobdir="jobdir") +c['schedulers'] = [s] +@end example + +Note that you must create the jobdir before telling the buildmaster to +use this configuration, otherwise you will get an error. Also remember +that the buildmaster must be able to read and write to the jobdir as +well. Be sure to watch the @file{twistd.log} file (@pxref{Logfiles}) +as you start using the jobdir, to make sure the buildmaster is happy +with it. + +To use the username/password form of authentication, create a +@code{Try_Userpass} instance instead. It takes the same +@code{builderNames} argument as the @code{Try_Jobdir} form, but +accepts an addtional @code{port} argument (to specify the TCP port to +listen on) and a @code{userpass} list of username/password pairs to +accept. Remember to use good passwords for this: the security of the +buildslave accounts depends upon it: + +@example +from buildbot.scheduler import Try_Userpass +s = Try_Userpass("try2", ["full-linux", "full-netbsd", "full-OSX"], + port=8031, userpass=[("alice","pw1"), ("bob", "pw2")] ) +c['schedulers'] = [s] +@end example + +Like most places in the buildbot, the @code{port} argument takes a +strports specification. See @code{twisted.application.strports} for +details. + + +@heading locating the master + +The @command{try} command needs to be told how to connect to the +@code{TryScheduler}, and must know which of the authentication +approaches described above is in use by the buildmaster. You specify +the approach by using @option{--connect=ssh} or @option{--connect=pb} +(or @code{try_connect = 'ssh'} or @code{try_connect = 'pb'} in +@file{.buildbot/options}). + +For the PB approach, the command must be given a @option{--master} +argument (in the form HOST:PORT) that points to TCP port that you +picked in the @code{Try_Userpass} scheduler. It also takes a +@option{--username} and @option{--passwd} pair of arguments that match +one of the entries in the buildmaster's @code{userpass} list. These +arguments can also be provided as @code{try_master}, +@code{try_username}, and @code{try_password} entries in the +@file{.buildbot/options} file. + +For the SSH approach, the command must be given @option{--tryhost}, +@option{--username}, and optionally @option{--password} (TODO: +really?) to get to the buildmaster host. It must also be given +@option{--trydir}, which points to the inlet directory configured +above. The trydir can be relative to the user's home directory, but +most of the time you will use an explicit path like +@file{~buildbot/project/trydir}. These arguments can be provided in +@file{.buildbot/options} as @code{try_host}, @code{try_username}, +@code{try_password}, and @code{try_dir}. + +In addition, the SSH approach needs to connect to a PBListener status +port, so it can retrieve and report the results of the build (the PB +approach uses the existing connection to retrieve status information, +so this step is not necessary). This requires a @option{--master} +argument, or a @code{masterstatus} entry in @file{.buildbot/options}, +in the form of a HOSTNAME:PORT string. + + +@heading choosing the Builders + +A trial build is performed on multiple Builders at the same time, and +the developer gets to choose which Builders are used (limited to a set +selected by the buildmaster admin with the TryScheduler's +@code{builderNames=} argument). The set you choose will depend upon +what your goals are: if you are concerned about cross-platform +compatibility, you should use multiple Builders, one from each +platform of interest. You might use just one builder if that platform +has libraries or other facilities that allow better test coverage than +what you can accomplish on your own machine, or faster test runs. + +The set of Builders to use can be specified with multiple +@option{--builder} arguments on the command line. It can also be +specified with a single @code{try_builders} option in +@file{.buildbot/options} that uses a list of strings to specify all +the Builder names: + +@example +try_builders = ["full-OSX", "full-win32", "full-linux"] +@end example + +If you are using the PB approach, you can get the names of the builders +that are configured for the try scheduler using the @code{get-builder-names} +argument: + +@example +buildbot try --get-builder-names --connect=pb --master=... --username=... --passwd=... +@end example + +@heading specifying the VC system + +The @command{try} command also needs to know how to take the +developer's current tree and extract the (revision, patch) +source-stamp pair. Each VC system uses a different process, so you +start by telling the @command{try} command which VC system you are +using, with an argument like @option{--vc=cvs} or @option{--vc=tla}. +This can also be provided as @code{try_vc} in +@file{.buildbot/options}. + +@c The order of this list comes from the end of scripts/tryclient.py + +The following names are recognized: @code{cvs} @code{svn} @code{baz} +@code{tla} @code{bzr} @code{hg} @code{darcs} @code{git} @code{p4} + + +@heading finding the top of the tree + +Some VC systems (notably CVS and SVN) track each directory +more-or-less independently, which means the @command{try} command +needs to move up to the top of the project tree before it will be able +to construct a proper full-tree patch. To accomplish this, the +@command{try} command will crawl up through the parent directories +until it finds a marker file. The default name for this marker file is +@file{.buildbot-top}, so when you are using CVS or SVN you should +@code{touch .buildbot-top} from the top of your tree before running +@command{buildbot try}. Alternatively, you can use a filename like +@file{ChangeLog} or @file{README}, since many projects put one of +these files in their top-most directory (and nowhere else). To set +this filename, use @option{--try-topfile=ChangeLog}, or set it in the +options file with @code{try_topfile = 'ChangeLog'}. + +You can also manually set the top of the tree with +@option{--try-topdir=~/trees/mytree}, or @code{try_topdir = +'~/trees/mytree'}. If you use @code{try_topdir}, in a +@file{.buildbot/options} file, you will need a separate options file +for each tree you use, so it may be more convenient to use the +@code{try_topfile} approach instead. + +Other VC systems which work on full projects instead of individual +directories (tla, baz, darcs, monotone, mercurial, git) do not require +@command{try} to know the top directory, so the @option{--try-topfile} +and @option{--try-topdir} arguments will be ignored. +@c is this true? I think I currently require topdirs all the time. + +If the @command{try} command cannot find the top directory, it will +abort with an error message. + +@heading determining the branch name + +Some VC systems record the branch information in a way that ``try'' +can locate it, in particular Arch (both @command{tla} and +@command{baz}). For the others, if you are using something other than +the default branch, you will have to tell the buildbot which branch +your tree is using. You can do this with either the @option{--branch} +argument, or a @option{try_branch} entry in the +@file{.buildbot/options} file. + +@heading determining the revision and patch + +Each VC system has a separate approach for determining the tree's base +revision and computing a patch. + +@table @code + +@item CVS + +@command{try} pretends that the tree is up to date. It converts the +current time into a @code{-D} time specification, uses it as the base +revision, and computes the diff between the upstream tree as of that +point in time versus the current contents. This works, more or less, +but requires that the local clock be in reasonably good sync with the +repository. + +@item SVN +@command{try} does a @code{svn status -u} to find the latest +repository revision number (emitted on the last line in the ``Status +against revision: NN'' message). It then performs an @code{svn diff +-rNN} to find out how your tree differs from the repository version, +and sends the resulting patch to the buildmaster. If your tree is not +up to date, this will result in the ``try'' tree being created with +the latest revision, then @emph{backwards} patches applied to bring it +``back'' to the version you actually checked out (plus your actual +code changes), but this will still result in the correct tree being +used for the build. + +@item baz +@command{try} does a @code{baz tree-id} to determine the +fully-qualified version and patch identifier for the tree +(ARCHIVE/VERSION--patch-NN), and uses the VERSION--patch-NN component +as the base revision. It then does a @code{baz diff} to obtain the +patch. + +@item tla +@command{try} does a @code{tla tree-version} to get the +fully-qualified version identifier (ARCHIVE/VERSION), then takes the +first line of @code{tla logs --reverse} to figure out the base +revision. Then it does @code{tla changes --diffs} to obtain the patch. + +@item bzr +@command{try} does a @code{bzr revision-info} to find the base revision, +then a @code{bzr diff -r$base..} to obtain the patch. + +@item Mercurial +@code{hg identify --debug} emits the full revision id (as opposed to +the common 12-char truncated) which is a SHA1 hash of the current +revision's contents. This is used as the base revision. +@code{hg diff} then provides the patch relative to that +revision. For @command{try} to work, your working directory must only +have patches that are available from the same remotely-available +repository that the build process' @code{source.Mercurial} will use. + +@item Perforce +@command{try} does a @code{p4 changes -m1 ...} to determine the latest +changelist and implicitly assumes that the local tree is synched to this +revision. This is followed by a @code{p4 diff -du} to obtain the patch. +A p4 patch differs sligtly from a normal diff. It contains full depot +paths and must be converted to paths relative to the branch top. To convert +the following restriction is imposed. The p4base (see @pxref{P4Source}) + is assumed to be @code{//depot} + +@item Darcs +@command{try} does a @code{darcs changes --context} to find the list +of all patches back to and including the last tag that was made. This text +file (plus the location of a repository that contains all these +patches) is sufficient to re-create the tree. Therefore the contents +of this ``context'' file @emph{are} the revision stamp for a +Darcs-controlled source tree. It then does a @code{darcs diff +-u} to compute the patch relative to that revision. + +@item Git +@code{git branch -v} lists all the branches available in the local +repository along with the revision ID it points to and a short summary +of the last commit. The line containing the currently checked out +branch begins with '* ' (star and space) while all the others start +with ' ' (two spaces). @command{try} scans for this line and extracts +the branch name and revision from it. Then it generates a diff against +the base revision. +@c TODO: I'm not sure if this actually works the way it's intended +@c since the extracted base revision might not actually exist in the +@c upstream repository. Perhaps we need to add a --remote option to +@c specify the remote tracking branch to generate a diff against. + +@c TODO: monotone +@end table + +@heading waiting for results + +If you provide the @option{--wait} option (or @code{try_wait = True} +in @file{.buildbot/options}), the @command{buildbot try} command will +wait until your changes have either been proven good or bad before +exiting. Unless you use the @option{--quiet} option (or +@code{try_quiet=True}), it will emit a progress message every 60 +seconds until the builds have completed. + +@menu +* try --diff:: +@end menu + +@node try --diff +@subsubsection try --diff + +Sometimes you might have a patch from someone else that you want to +submit to the buildbot. For example, a user may have created a patch +to fix some specific bug and sent it to you by email. You've inspected +the patch and suspect that it might do the job (and have at least +confirmed that it doesn't do anything evil). Now you want to test it +out. + +One approach would be to check out a new local tree, apply the patch, +run your local tests, then use ``buildbot try'' to run the tests on +other platforms. An alternate approach is to use the @command{buildbot +try --diff} form to have the buildbot test the patch without using a +local tree. + +This form takes a @option{--diff} argument which points to a file that +contains the patch you want to apply. By default this patch will be +applied to the TRUNK revision, but if you give the optional +@option{--baserev} argument, a tree of the given revision will be used +as a starting point instead of TRUNK. + +You can also use @command{buildbot try --diff=-} to read the patch +from stdin. + +Each patch has a ``patchlevel'' associated with it. This indicates the +number of slashes (and preceding pathnames) that should be stripped +before applying the diff. This exactly corresponds to the @option{-p} +or @option{--strip} argument to the @command{patch} utility. By +default @command{buildbot try --diff} uses a patchlevel of 0, but you +can override this with the @option{-p} argument. + +When you use @option{--diff}, you do not need to use any of the other +options that relate to a local tree, specifically @option{--vc}, +@option{--try-topfile}, or @option{--try-topdir}. These options will +be ignored. Of course you must still specify how to get to the +buildmaster (with @option{--connect}, @option{--tryhost}, etc). + + +@node Other Tools +@section Other Tools + +These tools are generally used by buildmaster administrators. + +@menu +* sendchange:: +* debugclient:: +@end menu + +@node sendchange +@subsection sendchange + +This command is used to tell the buildmaster about source changes. It +is intended to be used from within a commit script, installed on the +VC server. It requires that you have a PBChangeSource +(@pxref{PBChangeSource}) running in the buildmaster (by being set in +@code{c['change_source']}). + + +@example +buildbot sendchange --master @var{MASTERHOST}:@var{PORT} --username @var{USER} @var{FILENAMES..} +@end example + +The @code{master} and @code{username} arguments can also be given in the +options file (@pxref{.buildbot config directory}). There are other (optional) +arguments which can influence the @code{Change} that gets submitted: + +@table @code +@item --branch +(or option @code{branch}) This provides the (string) branch specifier. If +omitted, it defaults to None, indicating the ``default branch''. All files +included in this Change must be on the same branch. + +@item --category +(or option @code{category}) This provides the (string) category specifier. If +omitted, it defaults to None, indicating ``no category''. The category property +is used by Schedulers to filter what changes they listen to. + +@item --revision_number +This provides a (numeric) revision number for the change, used for VC systems +that use numeric transaction numbers (like Subversion). + +@item --revision +This provides a (string) revision specifier, for VC systems that use +strings (Arch would use something like patch-42 etc). + +@item --revision_file +This provides a filename which will be opened and the contents used as +the revision specifier. This is specifically for Darcs, which uses the +output of @command{darcs changes --context} as a revision specifier. +This context file can be a couple of kilobytes long, spanning a couple +lines per patch, and would be a hassle to pass as a command-line +argument. + +@item --property +This parameter is used to set a property on the Change generated by sendchange. +Properties are specified as a name:value pair, separated by a colon. You may +specify many properties by passing this parameter multiple times. + +@item --comments +This provides the change comments as a single argument. You may want +to use @option{--logfile} instead. + +@item --logfile +This instructs the tool to read the change comments from the given +file. If you use @code{-} as the filename, the tool will read the +change comments from stdin. +@end table + + +@node debugclient +@subsection debugclient + +@example +buildbot debugclient --master @var{MASTERHOST}:@var{PORT} --passwd @var{DEBUGPW} +@end example + +This launches a small Gtk+/Glade-based debug tool, connecting to the +buildmaster's ``debug port''. This debug port shares the same port +number as the slaveport (@pxref{Setting the slaveport}), but the +@code{debugPort} is only enabled if you set a debug password in the +buildmaster's config file (@pxref{Debug options}). The +@option{--passwd} option must match the @code{c['debugPassword']} +value. + +@option{--master} can also be provided in @file{.debug/options} by the +@code{master} key. @option{--passwd} can be provided by the +@code{debugPassword} key. @xref{.buildbot config directory}. + +The @code{Connect} button must be pressed before any of the other +buttons will be active. This establishes the connection to the +buildmaster. The other sections of the tool are as follows: + +@table @code +@item Reload .cfg +Forces the buildmaster to reload its @file{master.cfg} file. This is +equivalent to sending a SIGHUP to the buildmaster, but can be done +remotely through the debug port. Note that it is a good idea to be +watching the buildmaster's @file{twistd.log} as you reload the config +file, as any errors which are detected in the config file will be +announced there. + +@item Rebuild .py +(not yet implemented). The idea here is to use Twisted's ``rebuild'' +facilities to replace the buildmaster's running code with a new +version. Even if this worked, it would only be used by buildbot +developers. + +@item poke IRC +This locates a @code{words.IRC} status target and causes it to emit a +message on all the channels to which it is currently connected. This +was used to debug a problem in which the buildmaster lost the +connection to the IRC server and did not attempt to reconnect. + +@item Commit +This allows you to inject a Change, just as if a real one had been +delivered by whatever VC hook you are using. You can set the name of +the committed file and the name of the user who is doing the commit. +Optionally, you can also set a revision for the change. If the +revision you provide looks like a number, it will be sent as an +integer, otherwise it will be sent as a string. + +@item Force Build +This lets you force a Builder (selected by name) to start a build of +the current source tree. + +@item Currently +(obsolete). This was used to manually set the status of the given +Builder, but the status-assignment code was changed in an incompatible +way and these buttons are no longer meaningful. + +@end table + + +@node .buildbot config directory +@section .buildbot config directory + +Many of the @command{buildbot} tools must be told how to contact the +buildmaster that they interact with. This specification can be +provided as a command-line argument, but most of the time it will be +easier to set them in an ``options'' file. The @command{buildbot} +command will look for a special directory named @file{.buildbot}, +starting from the current directory (where the command was run) and +crawling upwards, eventually looking in the user's home directory. It +will look for a file named @file{options} in this directory, and will +evaluate it as a python script, looking for certain names to be set. +You can just put simple @code{name = 'value'} pairs in this file to +set the options. + +For a description of the names used in this file, please see the +documentation for the individual @command{buildbot} sub-commands. The +following is a brief sample of what this file's contents could be. + +@example +# for status-reading tools +masterstatus = 'buildbot.example.org:12345' +# for 'sendchange' or the debug port +master = 'buildbot.example.org:18990' +debugPassword = 'eiv7Po' +@end example + +Note carefully that the names in the @code{options} file usually do not match +the command-line option name. + +@table @code +@item masterstatus +Equivalent to @code{--master} for @ref{statuslog} and @ref{statusgui}, this +gives the location of the @code{client.PBListener} status port. + +@item master +Equivalent to @code{--master} for @ref{debugclient} and @ref{sendchange}. +This option is used for two purposes. It is the location of the +@code{debugPort} for @command{debugclient} and the location of the +@code{pb.PBChangeSource} for @command{sendchange}. Generally these are the +same port. + +@item debugPassword +Equivalent to @code{--passwd} for @ref{debugclient}. + +XXX Must match the value of @code{c['debugPassword']}, used to protect the +debug port, for the @command{debugclient} command. + +@item username +Equivalent to @code{--username} for the @ref{sendchange} command. + +@item branch +Equivalent to @code{--branch} for the @ref{sendchange} command. + +@item category +Equivalent to @code{--category} for the @ref{sendchange} command. + +@item try_connect +Equivalent to @code{--connect}, this specifies how the @ref{try} command should +deliver its request to the buildmaster. The currently accepted values are +``ssh'' and ``pb''. + +@item try_builders +Equivalent to @code{--builders}, specifies which builders should be used for +the @ref{try} build. + +@item try_vc +Equivalent to @code{--vc} for @ref{try}, this specifies the version control +system being used. + +@item try_branch +Equivlanent to @code{--branch}, this indicates that the current tree is on a non-trunk branch. + +@item try_topdir +@item try_topfile +Use @code{try_topdir}, equivalent to @code{--try-topdir}, to explicitly +indicate the top of your working tree, or @code{try_topfile}, equivalent to +@code{--try-topfile} to name a file that will only be found in that top-most +directory. + +@item try_host +@item try_username +@item try_dir +When @code{try_connect} is ``ssh'', the command will use @code{try_host} for +@code{--tryhost}, @code{try_username} for @code{--username}, and @code{try_dir} +for @code{--trydir}. Apologies for the confusing presence and absence of +'try'. + +@item try_username +@item try_password +@item try_master +Similarly, when @code{try_connect} is ``pb'', the command will pay attention to +@code{try_username} for @code{--username}, @code{try_password} for +@code{--passwd}, and @code{try_master} for @code{--master}. + +@item try_wait +@item masterstatus +@code{try_wait} and @code{masterstatus} (equivalent to @code{--wait} and +@code{master}, respectively) are used to ask the @ref{try} command to wait for +the requested build to complete. + +@end table + + + + diff --git a/docs/concepts.texinfo b/docs/concepts.texinfo new file mode 100644 index 00000000000..fbffaaf728c --- /dev/null +++ b/docs/concepts.texinfo @@ -0,0 +1,775 @@ +This chapter defines some of the basic concepts that the Buildbot +uses. You'll need to understand how the Buildbot sees the world to +configure it properly. + +@menu +* Version Control Systems:: +* Schedulers:: +* BuildSet:: +* BuildRequest:: +* Builder:: +* Users:: +* Build Properties:: +@end menu + +@node Version Control Systems +@section Version Control Systems + +@cindex Version Control + +These source trees come from a Version Control System of some kind. +CVS and Subversion are two popular ones, but the Buildbot supports +others. All VC systems have some notion of an upstream +@code{repository} which acts as a server@footnote{except Darcs, but +since the Buildbot never modifies its local source tree we can ignore +the fact that Darcs uses a less centralized model}, from which clients +can obtain source trees according to various parameters. The VC +repository provides source trees of various projects, for different +branches, and from various points in time. The first thing we have to +do is to specify which source tree we want to get. + +@menu +* Generalizing VC Systems:: +* Source Tree Specifications:: +* How Different VC Systems Specify Sources:: +* Attributes of Changes:: +@end menu + +@node Generalizing VC Systems +@subsection Generalizing VC Systems + +For the purposes of the Buildbot, we will try to generalize all VC +systems as having repositories that each provide sources for a variety +of projects. Each project is defined as a directory tree with source +files. The individual files may each have revisions, but we ignore +that and treat the project as a whole as having a set of revisions +(CVS is really the only VC system still in widespread use that has +per-file revisions.. everything modern has moved to atomic tree-wide +changesets). Each time someone commits a change to the project, a new +revision becomes available. These revisions can be described by a +tuple with two items: the first is a branch tag, and the second is +some kind of revision stamp or timestamp. Complex projects may have +multiple branch tags, but there is always a default branch. The +timestamp may be an actual timestamp (such as the -D option to CVS), +or it may be a monotonically-increasing transaction number (such as +the change number used by SVN and P4, or the revision number used by +Arch/Baz/Bazaar, or a labeled tag used in CVS)@footnote{many VC +systems provide more complexity than this: in particular the local +views that P4 and ClearCase can assemble out of various source +directories are more complex than we're prepared to take advantage of +here}. The SHA1 revision ID used by Monotone, Mercurial, and Git is +also a kind of revision stamp, in that it specifies a unique copy of +the source tree, as does a Darcs ``context'' file. + +When we aren't intending to make any changes to the sources we check out +(at least not any that need to be committed back upstream), there are two +basic ways to use a VC system: + +@itemize @bullet +@item +Retrieve a specific set of source revisions: some tag or key is used +to index this set, which is fixed and cannot be changed by subsequent +developers committing new changes to the tree. Releases are built from +tagged revisions like this, so that they can be rebuilt again later +(probably with controlled modifications). +@item +Retrieve the latest sources along a specific branch: some tag is used +to indicate which branch is to be used, but within that constraint we want +to get the latest revisions. +@end itemize + +Build personnel or CM staff typically use the first approach: the +build that results is (ideally) completely specified by the two +parameters given to the VC system: repository and revision tag. This +gives QA and end-users something concrete to point at when reporting +bugs. Release engineers are also reportedly fond of shipping code that +can be traced back to a concise revision tag of some sort. + +Developers are more likely to use the second approach: each morning +the developer does an update to pull in the changes committed by the +team over the last day. These builds are not easy to fully specify: it +depends upon exactly when you did a checkout, and upon what local +changes the developer has in their tree. Developers do not normally +tag each build they produce, because there is usually significant +overhead involved in creating these tags. Recreating the trees used by +one of these builds can be a challenge. Some VC systems may provide +implicit tags (like a revision number), while others may allow the use +of timestamps to mean ``the state of the tree at time X'' as opposed +to a tree-state that has been explicitly marked. + +The Buildbot is designed to help developers, so it usually works in +terms of @emph{the latest} sources as opposed to specific tagged +revisions. However, it would really prefer to build from reproducible +source trees, so implicit revisions are used whenever possible. + +@node Source Tree Specifications +@subsection Source Tree Specifications + +So for the Buildbot's purposes we treat each VC system as a server +which can take a list of specifications as input and produce a source +tree as output. Some of these specifications are static: they are +attributes of the builder and do not change over time. Others are more +variable: each build will have a different value. The repository is +changed over time by a sequence of Changes, each of which represents a +single developer making changes to some set of files. These Changes +are cumulative@footnote{Monotone's @emph{multiple heads} feature +violates this assumption of cumulative Changes, but in most situations +the changes don't occur frequently enough for this to be a significant +problem}. + +For normal builds, the Buildbot wants to get well-defined source trees +that contain specific Changes, and exclude other Changes that may have +occurred after the desired ones. We assume that the Changes arrive at +the buildbot (through one of the mechanisms described in @pxref{Change +Sources}) in the same order in which they are committed to the +repository. The Buildbot waits for the tree to become ``stable'' +before initiating a build, for two reasons. The first is that +developers frequently make multiple related commits in quick +succession, even when the VC system provides ways to make atomic +transactions involving multiple files at the same time. Running a +build in the middle of these sets of changes would use an inconsistent +set of source files, and is likely to fail (and is certain to be less +useful than a build which uses the full set of changes). The +tree-stable-timer is intended to avoid these useless builds that +include some of the developer's changes but not all. The second reason +is that some VC systems (i.e. CVS) do not provide repository-wide +transaction numbers, so that timestamps are the only way to refer to +a specific repository state. These timestamps may be somewhat +ambiguous, due to processing and notification delays. By waiting until +the tree has been stable for, say, 10 minutes, we can choose a +timestamp from the middle of that period to use for our source +checkout, and then be reasonably sure that any clock-skew errors will +not cause the build to be performed on an inconsistent set of source +files. + +The Schedulers always use the tree-stable-timer, with a timeout that +is configured to reflect a reasonable tradeoff between build latency +and change frequency. When the VC system provides coherent +repository-wide revision markers (such as Subversion's revision +numbers, or in fact anything other than CVS's timestamps), the +resulting Build is simply performed against a source tree defined by +that revision marker. When the VC system does not provide this, a +timestamp from the middle of the tree-stable period is used to +generate the source tree@footnote{this @code{checkoutDelay} defaults +to half the tree-stable timer, but it can be overridden with an +argument to the Source Step}. + +@node How Different VC Systems Specify Sources +@subsection How Different VC Systems Specify Sources + +For CVS, the static specifications are @code{repository} and +@code{module}. In addition to those, each build uses a timestamp (or +omits the timestamp to mean @code{the latest}) and @code{branch tag} +(which defaults to HEAD). These parameters collectively specify a set +of sources from which a build may be performed. + +@uref{http://subversion.tigris.org, Subversion} combines the +repository, module, and branch into a single @code{Subversion URL} +parameter. Within that scope, source checkouts can be specified by a +numeric @code{revision number} (a repository-wide +monotonically-increasing marker, such that each transaction that +changes the repository is indexed by a different revision number), or +a revision timestamp. When branches are used, the repository and +module form a static @code{baseURL}, while each build has a +@code{revision number} and a @code{branch} (which defaults to a +statically-specified @code{defaultBranch}). The @code{baseURL} and +@code{branch} are simply concatenated together to derive the +@code{svnurl} to use for the checkout. + +@uref{http://www.perforce.com/, Perforce} is similar. The server +is specified through a @code{P4PORT} parameter. Module and branch +are specified in a single depot path, and revisions are +depot-wide. When branches are used, the @code{p4base} and +@code{defaultBranch} are concatenated together to produce the depot +path. + +@uref{http://wiki.gnuarch.org/, Arch} and +@uref{http://bazaar.canonical.com/, Bazaar} specify a repository by +URL, as well as a @code{version} which is kind of like a branch name. +Arch uses the word @code{archive} to represent the repository. Arch +lets you push changes from one archive to another, removing the strict +centralization required by CVS and SVN. It retains the distinction +between repository and working directory that most other VC systems +use. For complex multi-module directory structures, Arch has a +built-in @code{build config} layer with which the checkout process has +two steps. First, an initial bootstrap checkout is performed to +retrieve a set of build-config files. Second, one of these files is +used to figure out which archives/modules should be used to populate +subdirectories of the initial checkout. + +Builders which use Arch and Bazaar therefore have a static archive +@code{url}, and a default ``branch'' (which is a string that specifies +a complete category--branch--version triple). Each build can have its +own branch (the category--branch--version string) to override the +default, as well as a revision number (which is turned into a +--patch-NN suffix when performing the checkout). + + +@uref{http://bazaar-vcs.org, Bzr} (which is a descendant of +Arch/Bazaar, and is frequently referred to as ``Bazaar'') has the same +sort of repository-vs-workspace model as Arch, but the repository data +can either be stored inside the working directory or kept elsewhere +(either on the same machine or on an entirely different machine). For +the purposes of Buildbot (which never commits changes), the repository +is specified with a URL and a revision number. + +The most common way to obtain read-only access to a bzr tree is via +HTTP, simply by making the repository visible through a web server +like Apache. Bzr can also use FTP and SFTP servers, if the buildslave +process has sufficient privileges to access them. Higher performance +can be obtained by running a special Bazaar-specific server. None of +these matter to the buildbot: the repository URL just has to match the +kind of server being used. The @code{repoURL} argument provides the +location of the repository. + +Branches are expressed as subdirectories of the main central +repository, which means that if branches are being used, the BZR step +is given a @code{baseURL} and @code{defaultBranch} instead of getting +the @code{repoURL} argument. + + +@uref{http://darcs.net/, Darcs} doesn't really have the +notion of a single master repository. Nor does it really have +branches. In Darcs, each working directory is also a repository, and +there are operations to push and pull patches from one of these +@code{repositories} to another. For the Buildbot's purposes, all you +need to do is specify the URL of a repository that you want to build +from. The build slave will then pull the latest patches from that +repository and build them. Multiple branches are implemented by using +multiple repositories (possibly living on the same server). + +Builders which use Darcs therefore have a static @code{repourl} which +specifies the location of the repository. If branches are being used, +the source Step is instead configured with a @code{baseURL} and a +@code{defaultBranch}, and the two strings are simply concatenated +together to obtain the repository's URL. Each build then has a +specific branch which replaces @code{defaultBranch}, or just uses the +default one. Instead of a revision number, each build can have a +``context'', which is a string that records all the patches that are +present in a given tree (this is the output of @command{darcs changes +--context}, and is considerably less concise than, e.g. Subversion's +revision number, but the patch-reordering flexibility of Darcs makes +it impossible to provide a shorter useful specification). + +@uref{http://selenic.com/mercurial, Mercurial} is like Darcs, in that +each branch is stored in a separate repository. The @code{repourl}, +@code{baseURL}, and @code{defaultBranch} arguments are all handled the +same way as with Darcs. The ``revision'', however, is the hash +identifier returned by @command{hg identify}. + +@uref{http://git.or.cz/, Git} also follows a decentralized model, and +each repository can have several branches and tags. The source Step is +configured with a static @code{repourl} which specifies the location +of the repository. In addition, an optional @code{branch} parameter +can be specified to check out code from a specific branch instead of +the default ``master'' branch. The ``revision'' is specified as a SHA1 +hash as returned by e.g. @command{git rev-parse}. No attempt is made +to ensure that the specified revision is actually a subset of the +specified branch. + + +@node Attributes of Changes +@subsection Attributes of Changes + +@heading Who + +Each Change has a @code{who} attribute, which specifies which +developer is responsible for the change. This is a string which comes +from a namespace controlled by the VC repository. Frequently this +means it is a username on the host which runs the repository, but not +all VC systems require this (Arch, for example, uses a fully-qualified +@code{Arch ID}, which looks like an email address, as does Darcs). +Each StatusNotifier will map the @code{who} attribute into something +appropriate for their particular means of communication: an email +address, an IRC handle, etc. + +@heading Files + +It also has a list of @code{files}, which are just the tree-relative +filenames of any files that were added, deleted, or modified for this +Change. These filenames are used by the @code{fileIsImportant} +function (in the Scheduler) to decide whether it is worth triggering a +new build or not, e.g. the function could use the following function +to only run a build if a C file were checked in: + +@example +def has_C_files(change): + for name in change.files: + if name.endswith(".c"): + return True + return False +@end example + +Certain BuildSteps can also use the list of changed files +to run a more targeted series of tests, e.g. the +@code{python_twisted.Trial} step can run just the unit tests that +provide coverage for the modified .py files instead of running the +full test suite. + +@heading Comments + +The Change also has a @code{comments} attribute, which is a string +containing any checkin comments. + +@heading Revision + +Each Change can have a @code{revision} attribute, which describes how +to get a tree with a specific state: a tree which includes this Change +(and all that came before it) but none that come after it. If this +information is unavailable, the @code{.revision} attribute will be +@code{None}. These revisions are provided by the ChangeSource, and +consumed by the @code{computeSourceRevision} method in the appropriate +@code{source.Source} class. + +@table @samp +@item CVS +@code{revision} is an int, seconds since the epoch +@item SVN +@code{revision} is an int, the changeset number (r%d) +@item Darcs +@code{revision} is a large string, the output of @code{darcs changes --context} +@item Mercurial +@code{revision} is a short string (a hash ID), the output of @code{hg identify} +@item Arch/Bazaar +@code{revision} is the full revision ID (ending in --patch-%d) +@item P4 +@code{revision} is an int, the transaction number +@item Git +@code{revision} is a short string (a SHA1 hash), the output of e.g. +@code{git rev-parse} +@end table + +@heading Branches + +The Change might also have a @code{branch} attribute. This indicates +that all of the Change's files are in the same named branch. The +Schedulers get to decide whether the branch should be built or not. + +For VC systems like CVS, Arch, Monotone, and Git, the @code{branch} +name is unrelated to the filename. (that is, the branch name and the +filename inhabit unrelated namespaces). For SVN, branches are +expressed as subdirectories of the repository, so the file's +``svnurl'' is a combination of some base URL, the branch name, and the +filename within the branch. (In a sense, the branch name and the +filename inhabit the same namespace). Darcs branches are +subdirectories of a base URL just like SVN. Mercurial branches are the +same as Darcs. + +@table @samp +@item CVS +branch='warner-newfeature', files=['src/foo.c'] +@item SVN +branch='branches/warner-newfeature', files=['src/foo.c'] +@item Darcs +branch='warner-newfeature', files=['src/foo.c'] +@item Mercurial +branch='warner-newfeature', files=['src/foo.c'] +@item Arch/Bazaar +branch='buildbot--usebranches--0', files=['buildbot/master.py'] +@item Git +branch='warner-newfeature', files=['src/foo.c'] +@end table + +@heading Build Properties + +A Change may have one or more properties attached to it, usually specified +through the Force Build form or @pxref{sendchange}. Properties are discussed +in detail in the @pxref{Build Properties} section. + +@heading Links + +@c TODO: who is using 'links'? how is it being used? + +Finally, the Change might have a @code{links} list, which is intended +to provide a list of URLs to a @emph{viewcvs}-style web page that +provides more detail for this Change, perhaps including the full file +diffs. + + +@node Schedulers +@section Schedulers + +@cindex Scheduler + +Each Buildmaster has a set of @code{Scheduler} objects, each of which +gets a copy of every incoming Change. The Schedulers are responsible +for deciding when Builds should be run. Some Buildbot installations +might have a single Scheduler, while others may have several, each for +a different purpose. + +For example, a ``quick'' scheduler might exist to give immediate +feedback to developers, hoping to catch obvious problems in the code +that can be detected quickly. These typically do not run the full test +suite, nor do they run on a wide variety of platforms. They also +usually do a VC update rather than performing a brand-new checkout +each time. You could have a ``quick'' scheduler which used a 30 second +timeout, and feeds a single ``quick'' Builder that uses a VC +@code{mode='update'} setting. + +A separate ``full'' scheduler would run more comprehensive tests a +little while later, to catch more subtle problems. This scheduler +would have a longer tree-stable-timer, maybe 30 minutes, and would +feed multiple Builders (with a @code{mode=} of @code{'copy'}, +@code{'clobber'}, or @code{'export'}). + +The @code{tree-stable-timer} and @code{fileIsImportant} decisions are +made by the Scheduler. Dependencies are also implemented here. +Periodic builds (those which are run every N seconds rather than after +new Changes arrive) are triggered by a special @code{Periodic} +Scheduler subclass. The default Scheduler class can also be told to +watch for specific branches, ignoring Changes on other branches. This +may be useful if you have a trunk and a few release branches which +should be tracked, but when you don't want to have the Buildbot pay +attention to several dozen private user branches. + +When the setup has multiple sources of Changes the @code{category} +can be used for @code{Scheduler} objects to filter out a subset +of the Changes. Note that not all change sources can attach a category. + +Some Schedulers may trigger builds for other reasons, other than +recent Changes. For example, a Scheduler subclass could connect to a +remote buildmaster and watch for builds of a library to succeed before +triggering a local build that uses that library. + +Each Scheduler creates and submits @code{BuildSet} objects to the +@code{BuildMaster}, which is then responsible for making sure the +individual @code{BuildRequests} are delivered to the target +@code{Builders}. + +@code{Scheduler} instances are activated by placing them in the +@code{c['schedulers']} list in the buildmaster config file. Each +Scheduler has a unique name. + + +@node BuildSet +@section BuildSet + +@cindex BuildSet + +A @code{BuildSet} is the name given to a set of Builds that all +compile/test the same version of the tree on multiple Builders. In +general, all these component Builds will perform the same sequence of +Steps, using the same source code, but on different platforms or +against a different set of libraries. + +The @code{BuildSet} is tracked as a single unit, which fails if any of +the component Builds have failed, and therefore can succeed only if +@emph{all} of the component Builds have succeeded. There are two kinds +of status notification messages that can be emitted for a BuildSet: +the @code{firstFailure} type (which fires as soon as we know the +BuildSet will fail), and the @code{Finished} type (which fires once +the BuildSet has completely finished, regardless of whether the +overall set passed or failed). + +A @code{BuildSet} is created with a @emph{source stamp} tuple of +(branch, revision, changes, patch), some of which may be None, and a +list of Builders on which it is to be run. They are then given to the +BuildMaster, which is responsible for creating a separate +@code{BuildRequest} for each Builder. + +There are a couple of different likely values for the +@code{SourceStamp}: + +@table @code +@item (revision=None, changes=[CHANGES], patch=None) +This is a @code{SourceStamp} used when a series of Changes have +triggered a build. The VC step will attempt to check out a tree that +contains CHANGES (and any changes that occurred before CHANGES, but +not any that occurred after them). + +@item (revision=None, changes=None, patch=None) +This builds the most recent code on the default branch. This is the +sort of @code{SourceStamp} that would be used on a Build that was +triggered by a user request, or a Periodic scheduler. It is also +possible to configure the VC Source Step to always check out the +latest sources rather than paying attention to the Changes in the +SourceStamp, which will result in same behavior as this. + +@item (branch=BRANCH, revision=None, changes=None, patch=None) +This builds the most recent code on the given BRANCH. Again, this is +generally triggered by a user request or Periodic build. + +@item (revision=REV, changes=None, patch=(LEVEL, DIFF, SUBDIR_ROOT)) +This checks out the tree at the given revision REV, then applies a +patch (using @code{patch -pLEVEL 'Account +Activity' and verifying EC2 is listed. +@end itemize + +@node Create an AMI +@subsubsection Create an AMI + +Now you need to create an AMI and configure the master. You may need to +run through this cycle a few times to get it working, but these instructions +should get you started. + +Creating an AMI is out of the scope of this document. The +@url{http://docs.amazonwebservices.com/AWSEC2/latest/GettingStartedGuide/,,EC2 Getting Started Guide} +is a good resource for this task. Here are a few additional hints. + +@itemize @bullet +@item +When an instance of the image starts, it needs to automatically start a +buildbot slave that connects to your master (to create a buildbot slave, +@pxref{Creating a buildslave}; to make a daemon, +@pxref{Launching the daemons}). + +@item +You may want to make an instance of the buildbot slave, configure it as a +standard buildslave in the master (i.e., not as a latent slave), and test and +debug it that way before you turn it into an AMI and convert to a latent +slave in the master. +@end itemize + +@node Configure the Master with an EC2LatentBuildSlave +@subsubsection Configure the Master with an EC2LatentBuildSlave + +Now let's assume you have an AMI that should work with the +EC2LatentBuildSlave. It's now time to set up your buildbot master +configuration. + +You will need some information from your AWS account: the "Access Key Id" and +the "Secret Access Key". If you've built the AMI yourself, you probably +already are familiar with these values. If you have not, and someone has +given you access to an AMI, these hints may help you find the necessary +values: + +@itemize @bullet +@item +While logged into your AWS account, find the "Access Identifiers" link (either +on the left, or via "Your Account" -> "Access Identifiers". + +@item +On the page, you'll see alphanumeric values for "Your Access Key Id:" and +"Your Secret Access Key:". Make a note of these. Later on, we'll call the +first one your "identifier" and the second one your "secret_identifier." +@end itemize + +When creating an EC2LatentBuildSlave in the buildbot master configuration, +the first three arguments are required. The name and password are the first +two arguments, and work the same as with normal buildslaves. The next +argument specifies the type of the EC2 virtual machine (available options as +of this writing include "m1.small", "m1.large", 'm1.xlarge", "c1.medium", +and "c1.xlarge"; see the EC2 documentation for descriptions of these +machines). + +Here is the simplest example of configuring an EC2 latent buildslave. It +specifies all necessary remaining values explicitly in the instantiation. + +@example +from buildbot.ec2buildslave import EC2LatentBuildSlave +c['slaves'] = [EC2LatentBuildSlave('bot1', 'sekrit', 'm1.large', + ami='ami-12345', + identifier='publickey', + secret_identifier='privatekey' + )] +@end example + +The "ami" argument specifies the AMI that the master should start. The +"identifier" argument specifies the AWS "Access Key Id," and the +"secret_identifier" specifies the AWS "Secret Access Key." Both the AMI and +the account information can be specified in alternate ways. + +Note that whoever has your identifier and secret_identifier values can request +AWS work charged to your account, so these values need to be carefully +protected. Another way to specify these access keys is to put them in a +separate file. You can then make the access privileges stricter for this +separate file, and potentially let more people read your main configuration +file. + +By default, you can make an .ec2 directory in the home folder of the user +running the buildbot master. In that directory, create a file called aws_id. +The first line of that file should be your access key id; the second line +should be your secret access key id. Then you can instantiate the build slave +as follows. + +@example +from buildbot.ec2buildslave import EC2LatentBuildSlave +c['slaves'] = [EC2LatentBuildSlave('bot1', 'sekrit', 'm1.large', + ami='ami-12345')] +@end example + +If you want to put the key information in another file, use the +"aws_id_file_path" initialization argument. + +Previous examples used a particular AMI. If the Buildbot master will be +deployed in a process-controlled environment, it may be convenient to +specify the AMI more flexibly. Rather than specifying an individual AMI, +specify one or two AMI filters. + +In all cases, the AMI that sorts last by its location (the S3 bucket and +manifest name) will be preferred. + +One available filter is to specify the acceptable AMI owners, by AWS account +number (the 12 digit number, usually rendered in AWS with hyphens like +"1234-5678-9012", should be entered as in integer). + +@example +from buildbot.ec2buildslave import EC2LatentBuildSlave +bot1 = EC2LatentBuildSlave('bot1', 'sekrit', 'm1.large', + valid_ami_owners=[11111111111, + 22222222222], + identifier='publickey', + secret_identifier='privatekey' + ) +@end example + +The other available filter is to provide a regular expression string that +will be matched against each AMI's location (the S3 bucket and manifest name). + +@example +from buildbot.ec2buildslave import EC2LatentBuildSlave +bot1 = EC2LatentBuildSlave( + 'bot1', 'sekrit', 'm1.large', + valid_ami_location_regex=r'buildbot\-.*/image.manifest.xml', + identifier='publickey', secret_identifier='privatekey') +@end example + +The regular expression can specify a group, which will be preferred for the +sorting. Only the first group is used; subsequent groups are ignored. + +@example +from buildbot.ec2buildslave import EC2LatentBuildSlave +bot1 = EC2LatentBuildSlave( + 'bot1', 'sekrit', 'm1.large', + valid_ami_location_regex=r'buildbot\-.*\-(.*)/image.manifest.xml', + identifier='publickey', secret_identifier='privatekey') +@end example + +If the group can be cast to an integer, it will be. This allows 10 to sort +after 1, for instance. + +@example +from buildbot.ec2buildslave import EC2LatentBuildSlave +bot1 = EC2LatentBuildSlave( + 'bot1', 'sekrit', 'm1.large', + valid_ami_location_regex=r'buildbot\-.*\-(\d+)/image.manifest.xml', + identifier='publickey', secret_identifier='privatekey') +@end example + +In addition to using the password as a handshake between the master and the +slave, you may want to use a firewall to assert that only machines from a +specific IP can connect as slaves. This is possible with AWS EC2 by using +the Elastic IP feature. To configure, generate a Elastic IP in AWS, and then +specify it in your configuration using the "elastic_ip" argument. + +@example +from buildbot.ec2buildslave import EC2LatentBuildSlave +c['slaves'] = [EC2LatentBuildSlave('bot1', 'sekrit', 'm1.large', + 'ami-12345', + identifier='publickey', + secret_identifier='privatekey', + elastic_ip='208.77.188.166' + )] +@end example + +The EC2LatentBuildSlave supports all other configuration from the standard +BuildSlave. The "missing_timeout" and "notify_on_missing" specify how long +to wait for an EC2 instance to attach before considering the attempt to have +failed, and email addresses to alert, respectively. "missing_timeout" +defaults to 20 minutes. + +The "build_wait_timeout" allows you to specify how long an EC2LatentBuildSlave +should wait after a build for another build before it shuts down the EC2 +instance. It defaults to 10 minutes. + +"keypair_name" and "security_name" allow you to specify different names for +these AWS EC2 values. They both default to "latent_buildbot_slave". + +@node Dangers with Latent Buildslaves +@subsection Dangers with Latent Buildslaves + +Any latent build slave that interacts with a for-fee service, such as the +EC2LatentBuildSlave, brings significant risks. As already identified, the +configuraton will need access to account information that, if obtained by a +criminal, can be used to charge services to your account. Also, bugs in the +buildbot software may lead to unnecessary charges. In particular, if the +master neglects to shut down an instance for some reason, a virtual machine +may be running unnecessarily, charging against your account. Manual and/or +automatic (e.g. nagios with a plugin using a library like boto) +double-checking may be appropriate. + +A comparitively trivial note is that currently if two instances try to attach +to the same latent buildslave, it is likely that the system will become +confused. This should not occur, unless, for instance, you configure a normal +build slave to connect with the authentication of a latent buildbot. If the +situation occurs, stop all attached instances and restart the master. + +@node Writing New Latent Buildslaves +@subsection Writing New Latent Buildslaves + +Writing a new latent buildslave should only require subclassing +@code{buildbot.buildslave.AbstractLatentBuildSlave} and implementing +start_instance and stop_instance. + +@example +def start_instance(self): + # responsible for starting instance that will try to connect with this + # master. Should return deferred. Problems should use an errback. The + # callback value can be None, or can be an iterable of short strings to + # include in the "substantiate success" status message, such as + # identifying the instance that started. + raise NotImplementedError + +def stop_instance(self, fast=False): + # responsible for shutting down instance. Return a deferred. If `fast`, + # we're trying to shut the master down, so callback as soon as is safe. + # Callback value is ignored. + raise NotImplementedError +@end example + +See @code{buildbot.ec2buildslave.EC2LatentBuildSlave} for an example, or see the +test example @code{buildbot.test_slaves.FakeLatentBuildSlave}. + +@node Defining Global Properties +@section Defining Global Properties +@bcindex c['properties'] +@cindex Properties + +The @code{'properties'} configuration key defines a dictionary +of properties that will be available to all builds started by the +buildmaster: + +@example +c['properties'] = @{ + 'Widget-version' : '1.2', + 'release-stage' : 'alpha' +@} +@end example + +@node Defining Builders +@section Defining Builders + +@bcindex c['builders'] + +The @code{c['builders']} key is a list of objects giving configuration for the +Builders. For more information, @xref{Builder}. The class definition for the +builder configuration is in @code{buildbot.config}. In the configuration file, +its use looks like: + +@example +from buildbot.config import BuilderConfig +c['builders'] = [ + BuilderConfig(name='quick', slavenames=['bot1', 'bot2'], factory=f_quick), + BuilderConfig(name='thorough', slavename='bot1', factory=f_thorough), +] +@end example + +The constructor takes the following keyword arguments: + +@table @code +@item name +This specifies the Builder's name, which is used in status +reports. + +@item slavename +@item slavenames +These arguments specify the buildslave or buildslaves that will be used by this +Builder. All slaves names must appear in the @code{c['slaves']} list. Each +buildslave can accomodate multiple Builders. The @code{slavenames} parameter +can be a list of names, while @code{slavename} can specify only one slave. + +@item factory +This is a @code{buildbot.process.factory.BuildFactory} instance which +controls how the build is performed. Full details appear in their own +section, @xref{Build Process}. Parameters like the location of the CVS +repository and the compile-time options used for the build are +generally provided as arguments to the factory's constructor. + +@end table + +Other optional keys may be set on each Builder: + +@table @code + +@item builddir +Specifies the name of a subdirectory (under the base directory) in which +everything related to this builder will be placed on the buildmaster. +This holds build status information. If not set, defaults to @code{name} +with some characters escaped. Each builder must have a unique build +directory. + +@item slavebuilddir +Specifies the name of a subdirectory (under the base directory) in which +everything related to this builder will be placed on the buildslave. +This is where checkouts, compiles, and tests are run. If not set, +defaults to @code{builddir}. If a slave is connected to multiple builders +that share the same @code{slavebuilddir}, make sure the slave is set to +run one build at a time or ensure this is fine to run multiple builds from +the same directory simultaneously. + +@item category +If provided, this is a string that identifies a category for the +builder to be a part of. Status clients can limit themselves to a +subset of the available categories. A common use for this is to add +new builders to your setup (for a new module, or for a new buildslave) +that do not work correctly yet and allow you to integrate them with +the active builders. You can put these new builders in a test +category, make your main status clients ignore them, and have only +private status clients pick them up. As soon as they work, you can +move them over to the active category. + +@item nextSlave +If provided, this is a function that controls which slave will be assigned +future jobs. The function is passed two arguments, the @code{Builder} +object which is assigning a new job, and a list of @code{BuildSlave} +objects. The function should return one of the @code{BuildSlave} +objects, or @code{None} if none of the available slaves should be +used. + +@item nextBuild +If provided, this is a function that controls which build request will be +handled next. The function is passed two arguments, the @code{Builder} +object which is assigning a new job, and a list of @code{BuildRequest} +objects of pending builds. The function should return one of the +@code{BuildRequest} objects, or @code{None} if none of the pending +builds should be started. + +@item locks +This argument specifies a list of locks that apply to this builder; @xref{Interlocks}. + +@item env +A Builder may be given a dictionary of environment variables in this parameter. +The variables are used in @pxref{ShellCommand} steps in builds created by this +builder. The environment variables will override anything in the buildslave's +environment. Variables passed directly to a @code{ShellCommand} will override +variables of the same name passed to the Builder. + +For example, if you a pool of identical slaves it is often easier to manage +variables like PATH from Buildbot rather than manually editing it inside of +the slaves' environment. + +@example +f = factory.BuildFactory +f.addStep(ShellCommand( + command=['bash', './configure'])) +f.addStep(Compile()) + +c['builders'] = [ + BuilderConfig(name='test', factory=f, + slavenames=['slave1', 'slave2', 'slave3', 'slave4'], + env=@{'PATH': '/opt/local/bin:/opt/app/bin:/usr/local/bin:/usr/bin'@}), +] +@end example + +@end table + + +@node Defining Status Targets +@section Defining Status Targets + +The Buildmaster has a variety of ways to present build status to +various users. Each such delivery method is a ``Status Target'' object +in the configuration's @code{status} list. To add status targets, you +just append more objects to this list: + +@bcindex c['status'] + +@example +c['status'] = [] + +from buildbot.status import html +c['status'].append(html.Waterfall(http_port=8010)) + +from buildbot.status import mail +m = mail.MailNotifier(fromaddr="buildbot@@localhost", + extraRecipients=["builds@@lists.example.com"], + sendToInterestedUsers=False) +c['status'].append(m) + +from buildbot.status import words +c['status'].append(words.IRC(host="irc.example.com", nick="bb", + channels=["#example"])) +@end example + +Status delivery has its own chapter, @xref{Status Delivery}, in which +all the built-in status targets are documented. + +@node Limiting Memory and Disk Usage +@section Limiting Memory and Disk Usage + +Buildbot stores historical information on disk in the form of "Pickle" files +and compressed logfiles. In a large installation, these can quickly consume +disk space, yet in many cases developers never consult this historical +information. Four configuration parameters control the "pruning" of various +data, shown here with their default values: + +@bcindex c['buildHorizon'] +@bcindex c['eventHorizon'] +@bcindex c['logHorizon'] +@bcindex c['buildCacheSize'] +@example +c['buildHorizon'] = 100 +c['eventHorizon'] = 50 +c['logHorizon'] = 40 + +c['buildCacheSize'] = 15 + +c['builders'] = [ + @{'name': 'test1', 'builddir': 'test1', 'factory': f, 'buildHorizon': 150@} +] +@end example + +The @code{buildHorizon} specifies the minimum number of builds for each builder +which should be kept on disk. The @code{eventHorizon} specifies the minumum +number of events to keep -- events mostly describe connections and +disconnections of slaves, and are seldom helpful to developers. The +@code{logHorizon} gives the minimum number of builds for which logs should be +maintained; this parameter must be less than @code{buildHorizon}. Builds older +than @code{logHorizon} but not older than @code{buildHorizon} will maintain +their overall status and the status of each step, but the logfiles will be +deleted. + +Finally, the @code{buildCacheSize} gives the number of builds for each builder +which are cached in memory. This number should be larger than the number of +builds required for commonly-used status displays (the waterfall or grid +views), so that those displays do not miss the cache on a refresh. + +@node Debug options +@section Debug options + + +@bcindex c['debugPassword'] +If you set @code{c['debugPassword']}, then you can connect to the +buildmaster with the diagnostic tool launched by @code{buildbot +debugclient MASTER:PORT}. From this tool, you can reload the config +file, manually force builds, and inject changes, which may be useful +for testing your buildmaster without actually commiting changes to +your repository (or before you have the Change Sources set up). The +debug tool uses the same port number as the slaves do: +@code{c['slavePortnum']}, and is authenticated with this password. + +@example +c['debugPassword'] = "debugpassword" +@end example + +@bcindex c['manhole'] +If you set @code{c['manhole']} to an instance of one of the classes in +@code{buildbot.manhole}, you can telnet or ssh into the buildmaster +and get an interactive Python shell, which may be useful for debugging +buildbot internals. It is probably only useful for buildbot +developers. It exposes full access to the buildmaster's account +(including the ability to modify and delete files), so it should not +be enabled with a weak or easily guessable password. + +There are three separate @code{Manhole} classes. Two of them use SSH, +one uses unencrypted telnet. Two of them use a username+password +combination to grant access, one of them uses an SSH-style +@file{authorized_keys} file which contains a list of ssh public keys. + +@table @code +@item manhole.AuthorizedKeysManhole +You construct this with the name of a file that contains one SSH +public key per line, just like @file{~/.ssh/authorized_keys}. If you +provide a non-absolute filename, it will be interpreted relative to +the buildmaster's base directory. + +@item manhole.PasswordManhole +This one accepts SSH connections but asks for a username and password +when authenticating. It accepts only one such pair. + + +@item manhole.TelnetManhole +This accepts regular unencrypted telnet connections, and asks for a +username/password pair before providing access. Because this +username/password is transmitted in the clear, and because Manhole +access to the buildmaster is equivalent to granting full shell +privileges to both the buildmaster and all the buildslaves (and to all +accounts which then run code produced by the buildslaves), it is +highly recommended that you use one of the SSH manholes instead. + +@end table + +@example +# some examples: +from buildbot import manhole +c['manhole'] = manhole.AuthorizedKeysManhole(1234, "authorized_keys") +c['manhole'] = manhole.PasswordManhole(1234, "alice", "mysecretpassword") +c['manhole'] = manhole.TelnetManhole(1234, "bob", "snoop_my_password_please") +@end example + +The @code{Manhole} instance can be configured to listen on a specific +port. You may wish to have this listening port bind to the loopback +interface (sometimes known as ``lo0'', ``localhost'', or 127.0.0.1) to +restrict access to clients which are running on the same host. + +@example +from buildbot.manhole import PasswordManhole +c['manhole'] = PasswordManhole("tcp:9999:interface=127.0.0.1","admin","passwd") +@end example + +To have the @code{Manhole} listen on all interfaces, use +@code{"tcp:9999"} or simply 9999. This port specification uses +@code{twisted.application.strports}, so you can make it listen on SSL +or even UNIX-domain sockets if you want. + +Note that using any Manhole requires that the TwistedConch package be +installed, and that you be using Twisted version 2.0 or later. + +The buildmaster's SSH server will use a different host key than the +normal sshd running on a typical unix host. This will cause the ssh +client to complain about a ``host key mismatch'', because it does not +realize there are two separate servers running on the same host. To +avoid this, use a clause like the following in your @file{.ssh/config} +file: + +@example +Host remotehost-buildbot + HostName remotehost + HostKeyAlias remotehost-buildbot + Port 9999 + # use 'user' if you use PasswordManhole and your name is not 'admin'. + # if you use AuthorizedKeysManhole, this probably doesn't matter. + User admin +@end example + + + diff --git a/docs/installation.texinfo b/docs/installation.texinfo new file mode 100644 index 00000000000..27df9fab023 --- /dev/null +++ b/docs/installation.texinfo @@ -0,0 +1,648 @@ +@menu +* Requirements:: +* Installing the code:: +* Creating a buildmaster:: +* Upgrading an Existing Buildmaster:: +* Creating a buildslave:: +* Launching the daemons:: +* Logfiles:: +* Shutdown:: +* Maintenance:: +* Troubleshooting:: +@end menu + +@node Requirements +@section Requirements + +At a bare minimum, you'll need the following (for both the buildmaster +and a buildslave): + +@itemize @bullet +@item +Python: http://www.python.org + +Buildbot requires python-2.3 or later, and is primarily developed +against python-2.4. It is also tested against python-2.5 . + +@item +Twisted: http://twistedmatrix.com + +Both the buildmaster and the buildslaves require Twisted-2.0.x or +later. It has been tested against all releases of Twisted up to +Twisted-2.5.0 (the most recent as of this writing). As always, the +most recent version is recommended. + +Twisted is delivered as a collection of subpackages. You'll need at +least "Twisted" (the core package), and you'll also want TwistedMail, +TwistedWeb, and TwistedWords (for sending email, serving a web status +page, and delivering build status via IRC, respectively). You might +also want TwistedConch (for the encrypted Manhole debug port). Note +that Twisted requires ZopeInterface to be installed as well. + +@end itemize + +Certain other packages may be useful on the system running the +buildmaster: + +@itemize @bullet +@item +CVSToys: http://purl.net/net/CVSToys + +If your buildmaster uses FreshCVSSource to receive change notification +from a cvstoys daemon, it will require CVSToys be installed (tested +with CVSToys-1.0.10). If the it doesn't use that source (i.e. if you +only use a mail-parsing change source, or the SVN notification +script), you will not need CVSToys. + +@end itemize + +And of course, your project's build process will impose additional +requirements on the buildslaves. These hosts must have all the tools +necessary to compile and test your project's source code. + + +@node Installing the code +@section Installing the code + +@cindex installation + +The Buildbot is installed using the standard python @code{distutils} +module. After unpacking the tarball, the process is: + +@example +python setup.py build +python setup.py install +@end example + +where the install step may need to be done as root. This will put the +bulk of the code in somewhere like +/usr/lib/python2.3/site-packages/buildbot . It will also install the +@code{buildbot} command-line tool in /usr/bin/buildbot. + +To test this, shift to a different directory (like /tmp), and run: + +@example +buildbot --version +@end example + +If it shows you the versions of Buildbot and Twisted, the install went +ok. If it says @code{no such command} or it gets an @code{ImportError} +when it tries to load the libaries, then something went wrong. +@code{pydoc buildbot} is another useful diagnostic tool. + +Windows users will find these files in other places. You will need to +make sure that python can find the libraries, and will probably find +it convenient to have @code{buildbot} on your PATH. + +If you wish, you can run the buildbot unit test suite like this: + +@example +PYTHONPATH=. trial buildbot.test +@end example + +This should run up to 192 tests, depending upon what VC tools you have +installed. On my desktop machine it takes about five minutes to +complete. Nothing should fail, a few might be skipped. If any of the +tests fail, you should stop and investigate the cause before +continuing the installation process, as it will probably be easier to +track down the bug early. + +If you cannot or do not wish to install the buildbot into a site-wide +location like @file{/usr} or @file{/usr/local}, you can also install +it into the account's home directory. Do the install command like +this: + +@example +python setup.py install --home=~ +@end example + +That will populate @file{~/lib/python} and create +@file{~/bin/buildbot}. Make sure this lib directory is on your +@code{PYTHONPATH}. + + +@node Creating a buildmaster +@section Creating a buildmaster + +As you learned earlier (@pxref{System Architecture}), the buildmaster +runs on a central host (usually one that is publically visible, so +everybody can check on the status of the project), and controls all +aspects of the buildbot system. Let us call this host +@code{buildbot.example.org}. + +You may wish to create a separate user account for the buildmaster, +perhaps named @code{buildmaster}. This can help keep your personal +configuration distinct from that of the buildmaster and is useful if +you have to use a mail-based notification system (@pxref{Change +Sources}). However, the Buildbot will work just fine with your regular +user account. + +You need to choose a directory for the buildmaster, called the +@code{basedir}. This directory will be owned by the buildmaster, which +will use configuration files therein, and create status files as it +runs. @file{~/Buildbot} is a likely value. If you run multiple +buildmasters in the same account, or if you run both masters and +slaves, you may want a more distinctive name like +@file{~/Buildbot/master/gnomovision} or +@file{~/Buildmasters/fooproject}. If you are using a separate user +account, this might just be @file{~buildmaster/masters/fooproject}. + +Once you've picked a directory, use the @command{buildbot +create-master} command to create the directory and populate it with +startup files: + +@example +buildbot create-master @var{basedir} +@end example + +You will need to create a configuration file (@pxref{Configuration}) +before starting the buildmaster. Most of the rest of this manual is +dedicated to explaining how to do this. A sample configuration file is +placed in the working directory, named @file{master.cfg.sample}, which +can be copied to @file{master.cfg} and edited to suit your purposes. + +(Internal details: This command creates a file named +@file{buildbot.tac} that contains all the state necessary to create +the buildmaster. Twisted has a tool called @code{twistd} which can use +this .tac file to create and launch a buildmaster instance. twistd +takes care of logging and daemonization (running the program in the +background). @file{/usr/bin/buildbot} is a front end which runs twistd +for you.) + +In addition to @file{buildbot.tac}, a small @file{Makefile.sample} is +installed. This can be used as the basis for customized daemon startup, +@xref{Launching the daemons}. + +@node Upgrading an Existing Buildmaster +@section Upgrading an Existing Buildmaster + +If you have just installed a new version of the Buildbot code, and you +have buildmasters that were created using an older version, you'll +need to upgrade these buildmasters before you can use them. The +upgrade process adds and modifies files in the buildmaster's base +directory to make it compatible with the new code. + +@example +buildbot upgrade-master @var{basedir} +@end example + +This command will also scan your @file{master.cfg} file for +incompatibilities (by loading it and printing any errors or deprecation +warnings that occur). Each buildbot release tries to be compatible +with configurations that worked cleanly (i.e. without deprecation +warnings) on the previous release: any functions or classes that are +to be removed will first be deprecated in a release, to give users a +chance to start using their replacement. + +The 0.7.6 release introduced the @file{public_html/} directory, which +contains @file{index.html} and other files served by the +@code{WebStatus} and @code{Waterfall} status displays. The +@code{upgrade-master} command will create these files if they do not +already exist. It will not modify existing copies, but it will write a +new copy in e.g. @file{index.html.new} if the new version differs from +the version that already exists. + +The @code{upgrade-master} command is idempotent. It is safe to run it +multiple times. After each upgrade of the buildbot code, you should +use @code{upgrade-master} on all your buildmasters. + + +@node Creating a buildslave +@section Creating a buildslave + +Typically, you will be adding a buildslave to an existing buildmaster, +to provide additional architecture coverage. The buildbot +administrator will give you several pieces of information necessary to +connect to the buildmaster. You should also be somewhat familiar with +the project being tested, so you can troubleshoot build problems +locally. + +The buildbot exists to make sure that the project's stated ``how to +build it'' process actually works. To this end, the buildslave should +run in an environment just like that of your regular developers. +Typically the project build process is documented somewhere +(@file{README}, @file{INSTALL}, etc), in a document that should +mention all library dependencies and contain a basic set of build +instructions. This document will be useful as you configure the host +and account in which the buildslave runs. + +Here's a good checklist for setting up a buildslave: + +@enumerate +@item +Set up the account + +It is recommended (although not mandatory) to set up a separate user +account for the buildslave. This account is frequently named +@code{buildbot} or @code{buildslave}. This serves to isolate your +personal working environment from that of the slave's, and helps to +minimize the security threat posed by letting possibly-unknown +contributors run arbitrary code on your system. The account should +have a minimum of fancy init scripts. + +@item +Install the buildbot code + +Follow the instructions given earlier (@pxref{Installing the code}). +If you use a separate buildslave account, and you didn't install the +buildbot code to a shared location, then you will need to install it +with @code{--home=~} for each account that needs it. + +@item +Set up the host + +Make sure the host can actually reach the buildmaster. Usually the +buildmaster is running a status webserver on the same machine, so +simply point your web browser at it and see if you can get there. +Install whatever additional packages or libraries the project's +INSTALL document advises. (or not: if your buildslave is supposed to +make sure that building without optional libraries still works, then +don't install those libraries). + +Again, these libraries don't necessarily have to be installed to a +site-wide shared location, but they must be available to your build +process. Accomplishing this is usually very specific to the build +process, so installing them to @file{/usr} or @file{/usr/local} is +usually the best approach. + +@item +Test the build process + +Follow the instructions in the INSTALL document, in the buildslave's +account. Perform a full CVS (or whatever) checkout, configure, make, +run tests, etc. Confirm that the build works without manual fussing. +If it doesn't work when you do it by hand, it will be unlikely to work +when the buildbot attempts to do it in an automated fashion. + +@item +Choose a base directory + +This should be somewhere in the buildslave's account, typically named +after the project which is being tested. The buildslave will not touch +any file outside of this directory. Something like @file{~/Buildbot} +or @file{~/Buildslaves/fooproject} is appropriate. + +@item +Get the buildmaster host/port, botname, and password + +When the buildbot admin configures the buildmaster to accept and use +your buildslave, they will provide you with the following pieces of +information: + +@itemize @bullet +@item +your buildslave's name +@item +the password assigned to your buildslave +@item +the hostname and port number of the buildmaster, i.e. buildbot.example.org:8007 +@end itemize + +@item +Create the buildslave + +Now run the 'buildbot' command as follows: + +@example +buildbot create-slave @var{BASEDIR} @var{MASTERHOST}:@var{PORT} @var{SLAVENAME} @var{PASSWORD} +@end example + +This will create the base directory and a collection of files inside, +including the @file{buildbot.tac} file that contains all the +information you passed to the @code{buildbot} command. + +@item +Fill in the hostinfo files + +When it first connects, the buildslave will send a few files up to the +buildmaster which describe the host that it is running on. These files +are presented on the web status display so that developers have more +information to reproduce any test failures that are witnessed by the +buildbot. There are sample files in the @file{info} subdirectory of +the buildbot's base directory. You should edit these to correctly +describe you and your host. + +@file{BASEDIR/info/admin} should contain your name and email address. +This is the ``buildslave admin address'', and will be visible from the +build status page (so you may wish to munge it a bit if +address-harvesting spambots are a concern). + +@file{BASEDIR/info/host} should be filled with a brief description of +the host: OS, version, memory size, CPU speed, versions of relevant +libraries installed, and finally the version of the buildbot code +which is running the buildslave. + +The optional @file{BASEDIR/info/access_uri} can specify a URI which will +connect a user to the machine. Many systems accept @code{ssh://hostname} URIs +for this purpose. + +If you run many buildslaves, you may want to create a single +@file{~buildslave/info} file and share it among all the buildslaves +with symlinks. + +@end enumerate + +@menu +* Buildslave Options:: +@end menu + +@node Buildslave Options +@subsection Buildslave Options + +There are a handful of options you might want to use when creating the +buildslave with the @command{buildbot create-slave DIR } +command. You can type @command{buildbot create-slave --help} for a summary. +To use these, just include them on the @command{buildbot create-slave} +command line, like this: + +@example +buildbot create-slave --umask=022 ~/buildslave buildmaster.example.org:42012 myslavename mypasswd +@end example + +@table @code +@item --usepty +This is a boolean flag that tells the buildslave whether to launch child +processes in a PTY or with regular pipes (the default) when the master does not +specify. This option is deprecated, as this particular parameter is better +specified on the master. + +@item --umask +This is a string (generally an octal representation of an integer) +which will cause the buildslave process' ``umask'' value to be set +shortly after initialization. The ``twistd'' daemonization utility +forces the umask to 077 at startup (which means that all files created +by the buildslave or its child processes will be unreadable by any +user other than the buildslave account). If you want build products to +be readable by other accounts, you can add @code{--umask=022} to tell +the buildslave to fix the umask after twistd clobbers it. If you want +build products to be @emph{writable} by other accounts too, use +@code{--umask=000}, but this is likely to be a security problem. + +@item --keepalive +This is a number that indicates how frequently ``keepalive'' messages +should be sent from the buildslave to the buildmaster, expressed in +seconds. The default (600) causes a message to be sent to the +buildmaster at least once every 10 minutes. To set this to a lower +value, use e.g. @code{--keepalive=120}. + +If the buildslave is behind a NAT box or stateful firewall, these +messages may help to keep the connection alive: some NAT boxes tend to +forget about a connection if it has not been used in a while. When +this happens, the buildmaster will think that the buildslave has +disappeared, and builds will time out. Meanwhile the buildslave will +not realize than anything is wrong. + +@item --maxdelay +This is a number that indicates the maximum amount of time the +buildslave will wait between connection attempts, expressed in +seconds. The default (300) causes the buildslave to wait at most 5 +minutes before trying to connect to the buildmaster again. + +@item --log-size +This is the size in bytes when to rotate the Twisted log files. + +@item --log-count +This is the number of log rotations to keep around. You can either +specify a number or @code{None} (the default) to keep all +@file{twistd.log} files around. + +@end table + + +@node Launching the daemons +@section Launching the daemons + +Both the buildmaster and the buildslave run as daemon programs. To +launch them, pass the working directory to the @code{buildbot} +command: + +@example +buildbot start @var{BASEDIR} +@end example + +The @var{BASEDIR} is option and can be omitted if the current directory +contains the buildbot configuration (the @file{buildbot.tac} file). + +@example +buildbot start +@end example + +This command will start the daemon and then return, so normally it +will not produce any output. To verify that the programs are indeed +running, look for a pair of files named @file{twistd.log} and +@file{twistd.pid} that should be created in the working directory. +@file{twistd.pid} contains the process ID of the newly-spawned daemon. + +When the buildslave connects to the buildmaster, new directories will +start appearing in its base directory. The buildmaster tells the slave +to create a directory for each Builder which will be using that slave. +All build operations are performed within these directories: CVS +checkouts, compiles, and tests. + +Once you get everything running, you will want to arrange for the +buildbot daemons to be started at boot time. One way is to use +@code{cron}, by putting them in a @@reboot crontab entry@footnote{this +@@reboot syntax is understood by Vixie cron, which is the flavor +usually provided with linux systems. Other unices may have a cron that +doesn't understand @@reboot}: + +@example +@@reboot buildbot start @var{BASEDIR} +@end example + +When you run @command{crontab} to set this up, remember to do it as +the buildmaster or buildslave account! If you add this to your crontab +when running as your regular account (or worse yet, root), then the +daemon will run as the wrong user, quite possibly as one with more +authority than you intended to provide. + +It is important to remember that the environment provided to cron jobs +and init scripts can be quite different that your normal runtime. +There may be fewer environment variables specified, and the PATH may +be shorter than usual. It is a good idea to test out this method of +launching the buildslave by using a cron job with a time in the near +future, with the same command, and then check @file{twistd.log} to +make sure the slave actually started correctly. Common problems here +are for @file{/usr/local} or @file{~/bin} to not be on your +@code{PATH}, or for @code{PYTHONPATH} to not be set correctly. +Sometimes @code{HOME} is messed up too. + +To modify the way the daemons are started (perhaps you want to set +some environment variables first, or perform some cleanup each time), +you can create a file named @file{Makefile.buildbot} in the base +directory. When the @file{buildbot} front-end tool is told to +@command{start} the daemon, and it sees this file (and +@file{/usr/bin/make} exists), it will do @command{make -f +Makefile.buildbot start} instead of its usual action (which involves +running @command{twistd}). When the buildmaster or buildslave is +installed, a @file{Makefile.sample} is created which implements the +same behavior as the the @file{buildbot} tool uses, so if you want to +customize the process, just copy @file{Makefile.sample} to +@file{Makefile.buildbot} and edit it as necessary. + +Some distributions may include conveniences to make starting buildbot +at boot time easy. For instance, with the default buildbot package in +Debian-based distributions, you may only need to modify +@code{/etc/default/buildbot} (see also @code{/etc/init.d/buildbot}, which +reads the configuration in @code{/etc/default/buildbot}). + +@node Logfiles +@section Logfiles + +@cindex logfiles + +While a buildbot daemon runs, it emits text to a logfile, named +@file{twistd.log}. A command like @code{tail -f twistd.log} is useful +to watch the command output as it runs. + +The buildmaster will announce any errors with its configuration file +in the logfile, so it is a good idea to look at the log at startup +time to check for any problems. Most buildmaster activities will cause +lines to be added to the log. + +@node Shutdown +@section Shutdown + +To stop a buildmaster or buildslave manually, use: + +@example +buildbot stop @var{BASEDIR} +@end example + +This simply looks for the @file{twistd.pid} file and kills whatever +process is identified within. + +At system shutdown, all processes are sent a @code{SIGKILL}. The +buildmaster and buildslave will respond to this by shutting down +normally. + +The buildmaster will respond to a @code{SIGHUP} by re-reading its +config file. Of course, this only works on unix-like systems with +signal support, and won't work on Windows. The following shortcut is +available: + +@example +buildbot reconfig @var{BASEDIR} +@end example + +When you update the Buildbot code to a new release, you will need to +restart the buildmaster and/or buildslave before it can take advantage +of the new code. You can do a @code{buildbot stop @var{BASEDIR}} and +@code{buildbot start @var{BASEDIR}} in quick succession, or you can +use the @code{restart} shortcut, which does both steps for you: + +@example +buildbot restart @var{BASEDIR} +@end example + +There are certain configuration changes that are not handled cleanly +by @code{buildbot reconfig}. If this occurs, @code{buildbot restart} +is a more robust tool to fully switch over to the new configuration. + +@code{buildbot restart} may also be used to start a stopped Buildbot +instance. This behaviour is useful when writing scripts that stop, start +and restart Buildbot. + +A buildslave may also be gracefully shutdown from the +@pxref{WebStatus} status plugin. This is useful to shutdown a +buildslave without interrupting any current builds. The buildmaster +will wait until the buildslave is finished all its current builds, and +will then tell the buildslave to shutdown. + +@node Maintenance +@section Maintenance + +It is a good idea to check the buildmaster's status page every once in +a while, to see if your buildslave is still online. Eventually the +buildbot will probably be enhanced to send you email (via the +@file{info/admin} email address) when the slave has been offline for +more than a few hours. + +If you find you can no longer provide a buildslave to the project, please +let the project admins know, so they can put out a call for a +replacement. + +The Buildbot records status and logs output continually, each time a +build is performed. The status tends to be small, but the build logs +can become quite large. Each build and log are recorded in a separate +file, arranged hierarchically under the buildmaster's base directory. +To prevent these files from growing without bound, you should +periodically delete old build logs. A simple cron job to delete +anything older than, say, two weeks should do the job. The only trick +is to leave the @file{buildbot.tac} and other support files alone, for +which find's @code{-mindepth} argument helps skip everything in the +top directory. You can use something like the following: + +@example +@@weekly cd BASEDIR && find . -mindepth 2 i-path './public_html/*' -prune -o -type f -mtime +14 -exec rm @{@} \; +@@weekly cd BASEDIR && find twistd.log* -mtime +14 -exec rm @{@} \; +@end example + +@node Troubleshooting +@section Troubleshooting + +Here are a few hints on diagnosing common problems. + +@menu +* Starting the buildslave:: +* Connecting to the buildmaster:: +* Forcing Builds:: +@end menu + +@node Starting the buildslave +@subsection Starting the buildslave + +Cron jobs are typically run with a minimal shell (@file{/bin/sh}, not +@file{/bin/bash}), and tilde expansion is not always performed in such +commands. You may want to use explicit paths, because the @code{PATH} +is usually quite short and doesn't include anything set by your +shell's startup scripts (@file{.profile}, @file{.bashrc}, etc). If +you've installed buildbot (or other python libraries) to an unusual +location, you may need to add a @code{PYTHONPATH} specification (note +that python will do tilde-expansion on @code{PYTHONPATH} elements by +itself). Sometimes it is safer to fully-specify everything: + +@example +@@reboot PYTHONPATH=~/lib/python /usr/local/bin/buildbot start /usr/home/buildbot/basedir +@end example + +Take the time to get the @@reboot job set up. Otherwise, things will work +fine for a while, but the first power outage or system reboot you have will +stop the buildslave with nothing but the cries of sorrowful developers to +remind you that it has gone away. + +@node Connecting to the buildmaster +@subsection Connecting to the buildmaster + +If the buildslave cannot connect to the buildmaster, the reason should +be described in the @file{twistd.log} logfile. Some common problems +are an incorrect master hostname or port number, or a mistyped bot +name or password. If the buildslave loses the connection to the +master, it is supposed to attempt to reconnect with an +exponentially-increasing backoff. Each attempt (and the time of the +next attempt) will be logged. If you get impatient, just manually stop +and re-start the buildslave. + +When the buildmaster is restarted, all slaves will be disconnected, +and will attempt to reconnect as usual. The reconnect time will depend +upon how long the buildmaster is offline (i.e. how far up the +exponential backoff curve the slaves have travelled). Again, +@code{buildbot stop @var{BASEDIR}; buildbot start @var{BASEDIR}} will +speed up the process. + +@node Forcing Builds +@subsection Forcing Builds + +From the buildmaster's main status web page, you can force a build to be run on +your build slave if you set the proper authorization options in your +@code{master.cfg} file @pxref{WebStatus Configuration Parameters}. + +Figure out which column is for a builder that runs on your slave, click on that +builder's name, and the page that comes up will have a "Force Build" button. +Fill in the form, hit the button, and a moment later you should see your +slave's @file{twistd.log} filling with commands being run. Using @code{pstree} +or @code{top} should also reveal the cvs/make/gcc/etc processes being run by +the buildslave. Note that the same web page should also show the @file{admin} +and @file{host} information files that you configured earlier. + + diff --git a/docs/introduction.texinfo b/docs/introduction.texinfo new file mode 100644 index 00000000000..7d62ab54664 --- /dev/null +++ b/docs/introduction.texinfo @@ -0,0 +1,313 @@ +@cindex introduction + +The BuildBot is a system to automate the compile/test cycle required by most +software projects to validate code changes. By automatically rebuilding and +testing the tree each time something has changed, build problems are +pinpointed quickly, before other developers are inconvenienced by the +failure. The guilty developer can be identified and harassed without human +intervention. By running the builds on a variety of platforms, developers +who do not have the facilities to test their changes everywhere before +checkin will at least know shortly afterwards whether they have broken the +build or not. Warning counts, lint checks, image size, compile time, and +other build parameters can be tracked over time, are more visible, and +are therefore easier to improve. + +The overall goal is to reduce tree breakage and provide a platform to +run tests or code-quality checks that are too annoying or pedantic for +any human to waste their time with. Developers get immediate (and +potentially public) feedback about their changes, encouraging them to +be more careful about testing before checkin. + +Features: + +@itemize @bullet +@item +run builds on a variety of slave platforms +@item +arbitrary build process: handles projects using C, Python, whatever +@item +minimal host requirements: python and Twisted +@item +slaves can be behind a firewall if they can still do checkout +@item +status delivery through web page, email, IRC, other protocols +@item +track builds in progress, provide estimated completion time +@item +flexible configuration by subclassing generic build process classes +@item +debug tools to force a new build, submit fake Changes, query slave status +@item +released under the GPL +@end itemize + +@menu +* History and Philosophy:: +* System Architecture:: +* Control Flow:: +@end menu + + +@node History and Philosophy +@section History and Philosophy + +@cindex Philosophy of operation + +The Buildbot was inspired by a similar project built for a development +team writing a cross-platform embedded system. The various components +of the project were supposed to compile and run on several flavors of +unix (linux, solaris, BSD), but individual developers had their own +preferences and tended to stick to a single platform. From time to +time, incompatibilities would sneak in (some unix platforms want to +use @code{string.h}, some prefer @code{strings.h}), and then the tree +would compile for some developers but not others. The buildbot was +written to automate the human process of walking into the office, +updating a tree, compiling (and discovering the breakage), finding the +developer at fault, and complaining to them about the problem they had +introduced. With multiple platforms it was difficult for developers to +do the right thing (compile their potential change on all platforms); +the buildbot offered a way to help. + +Another problem was when programmers would change the behavior of a +library without warning its users, or change internal aspects that +other code was (unfortunately) depending upon. Adding unit tests to +the codebase helps here: if an application's unit tests pass despite +changes in the libraries it uses, you can have more confidence that +the library changes haven't broken anything. Many developers +complained that the unit tests were inconvenient or took too long to +run: having the buildbot run them reduces the developer's workload to +a minimum. + +In general, having more visibility into the project is always good, +and automation makes it easier for developers to do the right thing. +When everyone can see the status of the project, developers are +encouraged to keep the tree in good working order. Unit tests that +aren't run on a regular basis tend to suffer from bitrot just like +code does: exercising them on a regular basis helps to keep them +functioning and useful. + +The current version of the Buildbot is additionally targeted at +distributed free-software projects, where resources and platforms are +only available when provided by interested volunteers. The buildslaves +are designed to require an absolute minimum of configuration, reducing +the effort a potential volunteer needs to expend to be able to +contribute a new test environment to the project. The goal is for +anyone who wishes that a given project would run on their favorite +platform should be able to offer that project a buildslave, running on +that platform, where they can verify that their portability code +works, and keeps working. + +@node System Architecture +@comment node-name, next, previous, up +@section System Architecture + +The Buildbot consists of a single @code{buildmaster} and one or more +@code{buildslaves}, connected in a star topology. The buildmaster +makes all decisions about what, when, and how to build. It sends +commands to be run on the build slaves, which simply execute the +commands and return the results. (certain steps involve more local +decision making, where the overhead of sending a lot of commands back +and forth would be inappropriate, but in general the buildmaster is +responsible for everything). + +The buildmaster is usually fed @code{Changes} by some sort of version control +system (@pxref{Change Sources}), which may cause builds to be run. As the +builds are performed, various status messages are produced, which are then sent +to any registered Status Targets (@pxref{Status Delivery}). + +@c @image{FILENAME, WIDTH, HEIGHT, ALTTEXT, EXTENSION} +@image{images/overview,,,Overview Diagram,} + +The buildmaster is configured and maintained by the ``buildmaster +admin'', who is generally the project team member responsible for +build process issues. Each buildslave is maintained by a ``buildslave +admin'', who do not need to be quite as involved. Generally slaves are +run by anyone who has an interest in seeing the project work well on +their favorite platform. + +@menu +* BuildSlave Connections:: +* Buildmaster Architecture:: +* Status Delivery Architecture:: +@end menu + +@node BuildSlave Connections +@subsection BuildSlave Connections + +The buildslaves are typically run on a variety of separate machines, +at least one per platform of interest. These machines connect to the +buildmaster over a TCP connection to a publically-visible port. As a +result, the buildslaves can live behind a NAT box or similar +firewalls, as long as they can get to buildmaster. The TCP connections +are initiated by the buildslave and accepted by the buildmaster, but +commands and results travel both ways within this connection. The +buildmaster is always in charge, so all commands travel exclusively +from the buildmaster to the buildslave. + +To perform builds, the buildslaves must typically obtain source code +from a CVS/SVN/etc repository. Therefore they must also be able to +reach the repository. The buildmaster provides instructions for +performing builds, but does not provide the source code itself. + +@image{images/slaves,,,BuildSlave Connections,} + +@node Buildmaster Architecture +@subsection Buildmaster Architecture + +The Buildmaster consists of several pieces: + +@image{images/master,,,BuildMaster Architecture,} + +@itemize @bullet + +@item +Change Sources, which create a Change object each time something is +modified in the VC repository. Most ChangeSources listen for messages +from a hook script of some sort. Some sources actively poll the +repository on a regular basis. All Changes are fed to the Schedulers. + +@item +Schedulers, which decide when builds should be performed. They collect +Changes into BuildRequests, which are then queued for delivery to +Builders until a buildslave is available. + +@item +Builders, which control exactly @emph{how} each build is performed +(with a series of BuildSteps, configured in a BuildFactory). Each +Build is run on a single buildslave. + +@item +Status plugins, which deliver information about the build results +through protocols like HTTP, mail, and IRC. + +@end itemize + +Each Builder is configured with a list of BuildSlaves that it will use +for its builds. These buildslaves are expected to behave identically: +the only reason to use multiple BuildSlaves for a single Builder is to +provide a measure of load-balancing. + +Within a single BuildSlave, each Builder creates its own SlaveBuilder +instance. These SlaveBuilders operate independently from each other. +Each gets its own base directory to work in. It is quite common to +have many Builders sharing the same buildslave. For example, there +might be two buildslaves: one for i386, and a second for PowerPC. +There may then be a pair of Builders that do a full compile/test run, +one for each architecture, and a lone Builder that creates snapshot +source tarballs if the full builders complete successfully. The full +builders would each run on a single buildslave, whereas the tarball +creation step might run on either buildslave (since the platform +doesn't matter when creating source tarballs). In this case, the +mapping would look like: + +@example +Builder(full-i386) -> BuildSlaves(slave-i386) +Builder(full-ppc) -> BuildSlaves(slave-ppc) +Builder(source-tarball) -> BuildSlaves(slave-i386, slave-ppc) +@end example + +and each BuildSlave would have two SlaveBuilders inside it, one for a +full builder, and a second for the source-tarball builder. + +Once a SlaveBuilder is available, the Builder pulls one or more +BuildRequests off its incoming queue. (It may pull more than one if it +determines that it can merge the requests together; for example, there +may be multiple requests to build the current HEAD revision). These +requests are merged into a single Build instance, which includes the +SourceStamp that describes what exact version of the source code +should be used for the build. The Build is then randomly assigned to a +free SlaveBuilder and the build begins. + +The behaviour when BuildRequests are merged can be customized, @pxref{Merging +BuildRequests}. + +@node Status Delivery Architecture +@subsection Status Delivery Architecture + +The buildmaster maintains a central Status object, to which various +status plugins are connected. Through this Status object, a full +hierarchy of build status objects can be obtained. + +@image{images/status,,,Status Delivery,} + +The configuration file controls which status plugins are active. Each +status plugin gets a reference to the top-level Status object. From +there they can request information on each Builder, Build, Step, and +LogFile. This query-on-demand interface is used by the html.Waterfall +plugin to create the main status page each time a web browser hits the +main URL. + +The status plugins can also subscribe to hear about new Builds as they +occur: this is used by the MailNotifier to create new email messages +for each recently-completed Build. + +The Status object records the status of old builds on disk in the +buildmaster's base directory. This allows it to return information +about historical builds. + +There are also status objects that correspond to Schedulers and +BuildSlaves. These allow status plugins to report information about +upcoming builds, and the online/offline status of each buildslave. + + +@node Control Flow +@comment node-name, next, previous, up +@section Control Flow + +A day in the life of the buildbot: + +@itemize @bullet + +@item +A developer commits some source code changes to the repository. A hook +script or commit trigger of some sort sends information about this +change to the buildmaster through one of its configured Change +Sources. This notification might arrive via email, or over a network +connection (either initiated by the buildmaster as it ``subscribes'' +to changes, or by the commit trigger as it pushes Changes towards the +buildmaster). The Change contains information about who made the +change, what files were modified, which revision contains the change, +and any checkin comments. + +@item +The buildmaster distributes this change to all of its configured +Schedulers. Any ``important'' changes cause the ``tree-stable-timer'' +to be started, and the Change is added to a list of those that will go +into a new Build. When the timer expires, a Build is started on each +of a set of configured Builders, all compiling/testing the same source +code. Unless configured otherwise, all Builds run in parallel on the +various buildslaves. + +@item +The Build consists of a series of Steps. Each Step causes some number +of commands to be invoked on the remote buildslave associated with +that Builder. The first step is almost always to perform a checkout of +the appropriate revision from the same VC system that produced the +Change. The rest generally perform a compile and run unit tests. As +each Step runs, the buildslave reports back command output and return +status to the buildmaster. + +@item +As the Build runs, status messages like ``Build Started'', ``Step +Started'', ``Build Finished'', etc, are published to a collection of +Status Targets. One of these targets is usually the HTML ``Waterfall'' +display, which shows a chronological list of events, and summarizes +the results of the most recent build at the top of each column. +Developers can periodically check this page to see how their changes +have fared. If they see red, they know that they've made a mistake and +need to fix it. If they see green, they know that they've done their +duty and don't need to worry about their change breaking anything. + +@item +If a MailNotifier status target is active, the completion of a build +will cause email to be sent to any developers whose Changes were +incorporated into this Build. The MailNotifier can be configured to +only send mail upon failing builds, or for builds which have just +transitioned from passing to failing. Other status targets can provide +similar real-time notification via different communication channels, +like IRC. + +@end itemize + + + diff --git a/docs/resources.texinfo b/docs/resources.texinfo new file mode 100644 index 00000000000..81840a17e7c --- /dev/null +++ b/docs/resources.texinfo @@ -0,0 +1,9 @@ +The Buildbot home page is @uref{http://buildbot.net/}. + +For configuration questions and general discussion, please use the +@code{buildbot-devel} mailing list. The subscription instructions and +archives are available at +@uref{http://lists.sourceforge.net/lists/listinfo/buildbot-devel} + +The #buildbot channel on Freenode's IRC servers hosts development discussion, and +often folks are available to answer questions there, as well. diff --git a/docs/status.texinfo b/docs/status.texinfo new file mode 100644 index 00000000000..9d7d1195a0c --- /dev/null +++ b/docs/status.texinfo @@ -0,0 +1,1054 @@ +More details are available in the docstrings for each class, use a +command like @code{pydoc buildbot.status.html.WebStatus} to see them. +Most status delivery objects take a @code{categories=} argument, which +can contain a list of ``category'' names: in this case, it will only +show status for Builders that are in one of the named categories. + +(implementor's note: each of these objects should be a +service.MultiService which will be attached to the BuildMaster object +when the configuration is processed. They should use +@code{self.parent.getStatus()} to get access to the top-level IStatus +object, either inside @code{startService} or later. They may call +@code{status.subscribe()} in @code{startService} to receive +notifications of builder events, in which case they must define +@code{builderAdded} and related methods. See the docstrings in +@file{buildbot/interfaces.py} for full details.) + +@menu +* WebStatus:: +* MailNotifier:: +* IRC Bot:: +* PBListener:: +* Writing New Status Plugins:: +@end menu + +@c @node Email Delivery +@c @subsection Email Delivery + +@c DOCUMENT THIS + + +@node WebStatus +@section WebStatus + +@cindex WebStatus +@stindex buildbot.status.web.baseweb.WebStatus + +The @code{buildbot.status.html.WebStatus} status target runs a small +web server inside the buildmaster. You can point a browser at this web +server and retrieve information about every build the buildbot knows +about, as well as find out what the buildbot is currently working on. + +The first page you will see is the ``Welcome Page'', which contains +links to all the other useful pages. This page is simply served from +the @file{public_html/index.html} file in the buildmaster's base +directory, where it is created by the @command{buildbot create-master} +command along with the rest of the buildmaster. + +The most complex resource provided by @code{WebStatus} is the +``Waterfall Display'', which shows a time-based chart of events. This +somewhat-busy display provides detailed information about all steps of +all recent builds, and provides hyperlinks to look at individual build +logs and source changes. By simply reloading this page on a regular +basis, you will see a complete description of everything the buildbot +is currently working on. + +A similar, but more developer-oriented display is the "Grid" display. This +arranges builds by SourceStamp (horizontal axis) and builder (vertical axis), +and can provide quick information as to which revisions are passing or failing +on which builders. + +There are also pages with more specialized information. For example, +there is a page which shows the last 20 builds performed by the +buildbot, one line each. Each line is a link to detailed information +about that build. By adding query arguments to the URL used to reach +this page, you can narrow the display to builds that involved certain +branches, or which ran on certain Builders. These pages are described +in great detail below. + + +When the buildmaster is created, a subdirectory named +@file{public_html/} is created in its base directory. By default, @code{WebStatus} +will serve files from this directory: for example, when a user points +their browser at the buildbot's @code{WebStatus} URL, they will see +the contents of the @file{public_html/index.html} file. Likewise, +@file{public_html/robots.txt}, @file{public_html/buildbot.css}, and +@file{public_html/favicon.ico} are all useful things to have in there. +The first time a buildmaster is created, the @file{public_html} +directory is populated with some sample files, which you will probably +want to customize for your own project. These files are all static: +the buildbot does not modify them in any way as it serves them to HTTP +clients. + +@example +from buildbot.status.html import WebStatus +c['status'].append(WebStatus(8080)) +@end example + +Note that the initial robots.txt file has Disallow lines for all of +the dynamically-generated buildbot pages, to discourage web spiders +and search engines from consuming a lot of CPU time as they crawl +through the entire history of your buildbot. If you are running the +buildbot behind a reverse proxy, you'll probably need to put the +robots.txt file somewhere else (at the top level of the parent web +server), and replace the URL prefixes in it with more suitable values. + +If you would like to use an alternative root directory, add the +@code{public_html=..} option to the @code{WebStatus} creation: + +@example +c['status'].append(WebStatus(8080, public_html="/var/www/buildbot")) +@end example + +In addition, if you are familiar with twisted.web @emph{Resource +Trees}, you can write code to add additional pages at places inside +this web space. Just use @code{webstatus.putChild} to place these +resources. + +The following section describes the special URLs and the status views +they provide. + + +@menu +* Buildbot Web Resources:: +* WebStatus Configuration Parameters:: +* XMLRPC server:: +@end menu + +@node Buildbot Web Resources +@subsection Buildbot Web Resources + +Certain URLs are ``magic'', and the pages they serve are created by +code in various classes in the @file{buildbot.status.web} package +instead of being read from disk. The most common way to access these +pages is for the buildmaster admin to write or modify the +@file{index.html} page to contain links to them. Of course other +project web pages can contain links to these buildbot pages as well. + +Many pages can be modified by adding query arguments to the URL. For +example, a page which shows the results of the most recent build +normally does this for all builders at once. But by appending +``?builder=i386'' to the end of the URL, the page will show only the +results for the ``i386'' builder. When used in this way, you can add +multiple ``builder='' arguments to see multiple builders. Remembering +that URL query arguments are separated @emph{from each other} with +ampersands, a URL that ends in ``?builder=i386&builder=ppc'' would +show builds for just those two Builders. + +The @code{branch=} query argument can be used on some pages. This +filters the information displayed by that page down to only the builds +or changes which involved the given branch. Use @code{branch=trunk} to +reference the trunk: if you aren't intentionally using branches, +you're probably using trunk. Multiple @code{branch=} arguments can be +used to examine multiple branches at once (so appending +@code{?branch=foo&branch=bar} to the URL will show builds involving +either branch). No @code{branch=} arguments means to show builds and +changes for all branches. + +Some pages may include the Builder name or the build number in the +main part of the URL itself. For example, a page that describes Build +#7 of the ``i386'' builder would live at @file{/builders/i386/builds/7}. + +The table below lists all of the internal pages and the URLs that can +be used to access them. + +@table @code + +@item /waterfall + +This provides a chronologically-oriented display of the activity of +all builders. It is the same display used by the Waterfall display. + +By adding one or more ``builder='' query arguments, the Waterfall is +restricted to only showing information about the given Builders. By +adding one or more ``branch='' query arguments, the display is +restricted to showing information about the given branches. In +addition, adding one or more ``category='' query arguments to the URL +will limit the display to Builders that were defined with one of the +given categories. + +A 'show_events=true' query argument causes the display to include +non-Build events, like slaves attaching and detaching, as well as +reconfiguration events. 'show_events=false' hides these events. The +default is to show them. + +By adding the 'failures_only=true' query argument, the Waterfall is +restricted to only showing information about the builders that +are currently failing. A builder is considered failing if the +last finished build was not successful, a step in the current +build(s) is failing, or if the builder is offline. + +The @code{last_time=}, @code{first_time=}, and @code{show_time=} +arguments will control what interval of time is displayed. The default +is to show the latest events, but these can be used to look at earlier +periods in history. The @code{num_events=} argument also provides a +limit on the size of the displayed page. + +The Waterfall has references to resources many of the other portions +of the URL space: @file{/builders} for access to individual builds, +@file{/changes} for access to information about source code changes, +etc. + +@item /grid + +This provides a chronologically oriented display of builders, by +revision. The builders are listed down the left side of the page, +and the revisions are listed across the top. + +By adding one ore more ``category='' arguments the grid will be +restricted to revisions in those categories. + +A ``width=N'' argument will limit the number of revisions shown to N, +defaulting to 5. + +A ``branch=BRANCHNAME'' argument will limit the grid to revisions on +branch BRANCHNAME. + +@item /tgrid + +The Transposed Grid is similar to the standard grid, but, as the name +implies, transposes the grid: the revisions are listed down the left side +of the page, and the build hosts are listed across the top. It accepts +the same query arguments. The exception being that instead of ``width'' +the argument is named ``length.'' + +@item /console + +EXPERIMENTAL: This provides a developer-oriented display of the the last +changes and how they affected the builders. + +It allows a developer to quickly see the status of each builder for the +first build including his or her change. A green box means that the change +succeeded for all the steps for a given builder. A red box means that +the changed introduced a new regression on a builder. An orange box +means that at least one of the test failed, but it was also failing +in the previous build, so it is not possible to see if there was any +regressions from this change. Finally a yellow box means that the test +is in progress. + +By adding one or more ``builder='' query arguments, the Console view is +restricted to only showing information about the given Builders. By +adding one or more ``branch='' query arguments, the display is +restricted to showing information about the given branches. In +addition, adding one or more ``category='' query arguments to the URL +will limit the display to Builders that were defined with one of the +given categories. + +By adding one or more ``name='' query arguments to the URL, the console view is +restricted to only showing changes made by the given users. + +NOTE: To use this page, your buildbot.css file in public_html +must be the one found in buildbot/status/web/extended.css. + +The console view is still in development. At this moment it supports +only the source control managers that have an integer based revision id, +like svn. It also has some issues with displaying multiple braches at the +same time. If you do have multiple branches, you should use the +``branch='' query argument. The @code{order_console_by_time} option may help +with non-integer VCs: + +@example +w = html.WebStatus(http_port=8080, order_console_by_time=True) +@end example + +@item /rss + +This provides a rss feed summarizing all failed builds. The same +query-arguments used by 'waterfall' can be added to filter the +feed output. + +@item /atom + +This provides an atom feed summarizing all failed builds. The same +query-arguments used by 'waterfall' can be added to filter the feed +output. + +@item /json + +This view provides quick access to Buildbot status information in a form that +is easiliy digested from other programs, including JavaScript. See +@code{/json/help} for detailed interactive documentation of the output formats +for this view. + +@item /buildstatus?builder=$BUILDERNAME&number=$BUILDNUM + +This displays a waterfall-like chronologically-oriented view of all the +steps for a given build number on a given builder. + +@item /builders/$BUILDERNAME + +This describes the given Builder, and provides buttons to force a +build. A @code{numbuilds=} argument will control how many build lines +are displayed (5 by default). + +@item /builders/$BUILDERNAME/builds/$BUILDNUM + +This describes a specific Build. + +@item /builders/$BUILDERNAME/builds/$BUILDNUM/steps/$STEPNAME + +This describes a specific BuildStep. + +@item /builders/$BUILDERNAME/builds/$BUILDNUM/steps/$STEPNAME/logs/$LOGNAME + +This provides an HTML representation of a specific logfile. + +@item /builders/$BUILDERNAME/builds/$BUILDNUM/steps/$STEPNAME/logs/$LOGNAME/text + +This returns the logfile as plain text, without any HTML coloring +markup. It also removes the ``headers'', which are the lines that +describe what command was run and what the environment variable +settings were like. This maybe be useful for saving to disk and +feeding to tools like 'grep'. + +@item /changes + +This provides a brief description of the ChangeSource in use +(@pxref{Change Sources}). + +@item /changes/NN + +This shows detailed information about the numbered Change: who was the +author, what files were changed, what revision number was represented, +etc. + +@item /buildslaves + +This summarizes each BuildSlave, including which Builders are +configured to use it, whether the buildslave is currently connected or +not, and host information retrieved from the buildslave itself. + +@item /one_line_per_build + +This page shows one line of text for each build, merging information +from all Builders@footnote{Apparently this is the same way +http://buildd.debian.org displays build status}. Each line specifies +the name of the Builder, the number of the Build, what revision it +used, and a summary of the results. Successful builds are in green, +while failing builds are in red. The date and time of the build are +added to the right-hand edge of the line. The lines are ordered by +build finish timestamp. + +One or more @code{builder=} or @code{branch=} arguments can be used to +restrict the list. In addition, a @code{numbuilds=} argument will +control how many lines are displayed (20 by default). + +@item /builders + +This page shows a small table, with one box for each Builder, +containing the results of the most recent Build. It does not show the +individual steps, or the current status. This is a simple summary of +buildbot status: if this page is green, then all tests are passing. + +As with @code{/one_line_per_build}, this page will also honor +@code{builder=} and @code{branch=} arguments. + +@item /about + +This page gives a brief summary of the Buildbot itself: software +version, versions of some libraries that the Buildbot depends upon, +etc. It also contains a link to the buildbot.net home page. + +@end table + +There are also a set of web-status resources that are intended for use +by other programs, rather than humans. + +@table @code + +@item /xmlrpc + +This runs an XML-RPC server which can be used to query status +information about various builds. See @ref{XMLRPC server} for more +details. + +@end table + +@node WebStatus Configuration Parameters +@subsection WebStatus Configuration Parameters + +@heading HTTP Connection + +The most common way to run a @code{WebStatus} is on a regular TCP +port. To do this, just pass in the TCP port number when you create the +@code{WebStatus} instance; this is called the @code{http_port} argument: + +@example +from buildbot.status.html import WebStatus +c['status'].append(http_port=WebStatus(http_port=8080)) +@end example + +The @code{http_port} argument is actually a ``strports specification'' for the +port that the web server should listen on. This can be a simple port number, or +a string like @code{http_port="tcp:8080:interface=127.0.0.1"} (to limit +connections to the loopback interface, and therefore to clients running on the +same host)@footnote{It may even be possible to provide SSL access by using a +specification like @code{"ssl:12345:privateKey=mykey.pen:certKey=cert.pem"}, +but this is completely untested}. + +If instead (or in addition) you provide the @code{distrib_port} +argument, a twisted.web distributed server will be started either on a +TCP port (if @code{distrib_port} is like @code{"tcp:12345"}) or more +likely on a UNIX socket (if @code{distrib_port} is like +@code{"unix:/path/to/socket"}). + +The @code{public_html} option gives the path to a regular directory of HTML +files that will be displayed alongside the various built-in URLs buildbot +supplies. This is most often used to supply CSS files (@code{/buildbot.css}) +and a top-level navigational file (@code{/index.html}), but can also serve any +other files required - even build results! + +@heading Authorization + +The buildbot web status is, by default, read-only. It displays lots of +information, but users are not allowed to affect the operation of the +buildmaster. However, there are a number of supported activities that can +be enabled, and Buildbot can also perform rudimentary username/password +authentication. The actions are: + +@table @code + +@item forceBuild + +force a particular builder to begin building, optionally with a specific revision, branch, etc. + +@item forceAllBuilds + +force @i{all} builders to start building + +@item pingBuilder + +"ping" a builder's buildslaves to check that they are alive + +@item gracefulShutdown + +gracefully shut down a slave when it is finished with its current build + +@item stopBuild + +stop a running build + +@item stopAllBuilds + +stop all running builds + +@item cancelPendingBuild + +cancel a build that has not yet started + +@end table + +For each of these actions, you can configure buildbot to never allow the +action, always allow the action, allow the action to any authenticated user, or +check with a function of your creation to determine whether the action is OK. + +This is all configured with the @code{Authz} class: + +@example +from buildbot.status.html import WebStatus +from buildbot.status.web.authz import Authz +authz = Authz( + forceBuild=True, + stopBuild=True) +c['status'].append(http_port=WebStatus(http_port=8080, authz=authz)) +@end example + +Each of the actions listed above is an option to @code{Authz}. You can specify +@code{False} (the default) to prohibit that action or @code{True} to enable it. + +@heading Authentication + +If you do not wish to allow strangers to perform actions, but do want +developers to have such access, you will need to add some authentication +support. Pass an instance of @code{status.web.auth.IAuth} as a @code{auth} +keyword argument to @code{Authz}, and specify the action as @code{"auth"}. + +@example +from buildbot.status.html import WebStatus +from buildbot.status.web.authz import Authz +from buildbot.status.web.auth import BasicAuth +users = [('bob', 'secret-pass'), ('jill', 'super-pass')] +authz = Authz(auth=BasicAuth(users), + forceBuild='auth', # only authenticated users + pingBuilder=True, # but anyone can do this +) +c['status'].append(WebStatus(http_port=8080, authz=authz)) +# or +from buildbot.status.web.auth import HTPasswdAuth +auth = (HTPasswdAuth('/path/to/htpasswd')) +@end example + +The class @code{BasicAuth} implements a basic authentication mechanism using a +list of user/password tuples provided from the configuration file. The class +@code{HTPasswdAuth} implements an authentication against an @file{.htpasswd} +file. + +If you need still-more flexibility, pass a function for the authentication +action. That function will be called with an authenticated username and some +action-specific arguments, and should return true if the action is authorized. + +@example +def canForceBuild(username, builder_status): + if builder_status.getName() == 'smoketest': + return True # any authenticated user can run smoketest + elif username == 'releng': + return True # releng can force whatever they want + else: + return False # otherwise, no way. + +authz = Authz(auth=BasicAuth(users), + forceBuild=canForceBuild) +@end example + +The @code{forceBuild} and @code{pingBuilder} actions both supply a +BuilderStatus object. The @code{stopBuild} action supplies a BuildStatus +object. The @code{cancelPendingBuild} action supplies a BuildRequest. The +remainder do not supply any extra arguments. + +@heading Miscellaneous Options + +The @code{changecommentlink} argument can be used to create links to +ticket-ids from change comments (i.e. #123). The argument should +be a tuple of three strings where the first element is a regex that +searches for strings (with match groups), the second is a replace-string +that, when substituted with \1 etc, yields the URL and the third +is the title attribute of the link. +(The @code{} is added by the system.) +So, for Trac tickets (#42, etc): +@code{changecommentlink(r"#(\d+)", r"http://buildbot.net/trac/ticket/\1", r"Ticket \g<0>")} . + +The @code{revlink} is similarly used to create links from revision +ids to a web-view of your source control system. The Use %s to insert +the revision id in the url. I.e. for Buildbot on github: +@code{revlink='http://github.com/djmitche/buildbot/tree/%s'} +(The revision id will be URL encoded before inserted in the replacement string) + +@heading Display-Specific Options + +The @code{order_console_by_time} option affects the rendering of the console; +see the description of the console above. + +The @code{numbuilds} option determines the number of builds that most status +displays will show. It can usually be overriden in the URL, e.g., +@code{?numbuilds=13}. + +The @code{num_events} option gives the default number of events that the +waterfall will display. The @code{num_events_max} gives the maximum number of +events displayed, even if the web browser requests more. + +@node XMLRPC server +@subsection XMLRPC server + +When using WebStatus, the buildbot runs an XML-RPC server at +@file{/xmlrpc} that can be used by other programs to query build +status. The following table lists the methods that can be invoked +using this interface. + +@table @code +@item getAllBuildsInInterval(start, stop) + +Return a list of builds that have completed after the 'start' +timestamp and before the 'stop' timestamp. This looks at all Builders. + +The timestamps are integers, interpreted as standard unix timestamps +(seconds since epoch). + +Each Build is returned as a tuple in the form: @code{(buildername, +buildnumber, build_end, branchname, revision, results, text)} + +The buildnumber is an integer. 'build_end' is an integer (seconds +since epoch) specifying when the build finished. + +The branchname is a string, which may be an empty string to indicate +None (i.e. the default branch). The revision is a string whose meaning +is specific to the VC system in use, and comes from the 'got_revision' +build property. The results are expressed as a string, one of +('success', 'warnings', 'failure', 'exception'). The text is a list of +short strings that ought to be joined by spaces and include slightly +more data about the results of the build. + +@item getBuild(builder_name, build_number) + +Return information about a specific build. + +This returns a dictionary (aka ``struct'' in XMLRPC terms) with +complete information about the build. It does not include the contents +of the log files, but it has just about everything else. + +@end table + +@node MailNotifier +@section MailNotifier + +@cindex email +@cindex mail +@stindex buildbot.status.mail.MailNotifier + +The buildbot can also send email when builds finish. The most common +use of this is to tell developers when their change has caused the +build to fail. It is also quite common to send a message to a mailing +list (usually named ``builds'' or similar) about every build. + +The @code{MailNotifier} status target is used to accomplish this. You +configure it by specifying who mail should be sent to, under what +circumstances mail should be sent, and how to deliver the mail. It can +be configured to only send out mail for certain builders, and only +send messages when the build fails, or when the builder transitions +from success to failure. It can also be configured to include various +build logs in each message. + + +By default, the message will be sent to the Interested Users list +(@pxref{Doing Things With Users}), which includes all developers who +made changes in the build. You can add additional recipients with the +extraRecipients argument. + +Each MailNotifier sends mail to a single set of recipients. To send +different kinds of mail to different recipients, use multiple +MailNotifiers. + +The following simple example will send an email upon the completion of +each build, to just those developers whose Changes were included in +the build. The email contains a description of the Build, its results, +and URLs where more information can be obtained. + +@example +from buildbot.status.mail import MailNotifier +mn = MailNotifier(fromaddr="buildbot@@example.org", lookup="example.org") +c['status'].append(mn) +@end example + +To get a simple one-message-per-build (say, for a mailing list), use +the following form instead. This form does not send mail to individual +developers (and thus does not need the @code{lookup=} argument, +explained below), instead it only ever sends mail to the ``extra +recipients'' named in the arguments: + +@example +mn = MailNotifier(fromaddr="buildbot@@example.org", + sendToInterestedUsers=False, + extraRecipients=['listaddr@@example.org']) +@end example + +In some cases it is desirable to have different information then what is +provided in a standard MailNotifier message. For this purpose MailNotifier +provides the argument @code{messageFormatter} (a function) which allows for the +creation of messages with unique content. + +For example, if only short emails are desired (e.g., for delivery to phones) + +@example +from buildbot.status.builder import Results +def messageFormatter(mode, name, build, results, master_status): + result = Results[results] + + text = list() + text.append("STATUS: %s" % result.title()) + return @{ + 'body' : "\n".join(text), + 'type' : 'plain' + @} + +mn = MailNotifier(fromaddr="buildbot@@example.org", + sendToInterestedUsers=False, + mode='problem', + extraRecipients=['listaddr@@example.org'], + messageFormatter=messageFormatter) +@end example + +Another example of a function delivering a customized html email +containing the last 80 lines of logs of a failing build step is given +below: + +@example +from buildbot.status.builder import Results + +def message_formatter(mode, name, build, results, master_status): + """Provide a customized message to BuildBot's MailNotifier. + + The last 80 lines of the log are provided as well as the changes + relevant to the build. Message content is formatted as html. + """ + result = Results[results] + + limit_lines = 80 + text = list() + text.append('

Build status: %s

' % result.upper()) + text.append("Buildslave for this Build: %s" % build.getSlavename()) + text.append('
') + if master_status.getURLForThing(build): + text.append('Complete logs for all build steps: %s' + % (master_status.getURLForThing(build), + master_status.getURLForThing(build)) + ) + text.append('
') + text.append("Build Reason: %s" % build.getReason()) + text.append('
') + + source = "" + ss = build.getSourceStamp() + if ss.branch: + source += "[branch %s] " % ss.branch + if ss.revision: + source += ss.revision + else: + source += "HEAD" + if ss.patch: + source += " (plus patch)" + text.append("Build Source Stamp: %s" % source) + text.append('
') + text.append("Blamelist: %s" % ",".join(build.getResponsibleUsers())) + if ss.changes: + text.append('

Recent Changes:

') + text.extend([c.asHTML() for c in ss.changes]) + + logs = list() + for log in build.getLogs(): + log_name = "%s.%s" % (log.getStep().getName(), log.getName()) + log_status, dummy = log.getStep().getResults() + log_body = log.getText().splitlines() # Note: can be VERY LARGE + log_url = '%s/steps/%s/logs/%s' % (master_status.getURLForThing(build), + log.getStep().getName(), + log.getName()) + logs.append((log_name, log_url, log_body, log_status)) + + name, url, content, logstatus = logs[-1] + + text.append('Detailed log of last build step: %s' + % (url, url)) + text.append('
') + text.append('

Last %d lines of "%s":

' % (limit_lines, name)) + text.append('

') + text.append('
'.join([line for line in + content[len(content)-limit_lines:]])) + text.append('

') + text.append('

') + text.append('-The BuildBot') + return @{ + 'body': "\n".join(text), + 'type': 'html' + @} + +mn = MailNotifier(fromaddr="buildbot@@example.org", + sendToInterestedUsers=False, + mode='failing', + extraRecipients=['listaddr@@example.org'], + messageFormatter=message_formatter) +@end example + + +@heading MailNotifier arguments + +@table @code +@item fromaddr +The email address to be used in the 'From' header. + +@item sendToInterestedUsers +(boolean). If True (the default), send mail to all of the Interested +Users. If False, only send mail to the extraRecipients list. + +@item extraRecipients +(list of strings). A list of email addresses to which messages should +be sent (in addition to the InterestedUsers list, which includes any +developers who made Changes that went into this build). It is a good +idea to create a small mailing list and deliver to that, then let +subscribers come and go as they please. + +@item subject +(string). A string to be used as the subject line of the message. +@code{%(builder)s} will be replaced with the name of the builder which +provoked the message. + +@item mode +(string). Default to 'all'. One of: +@table @code +@item all +Send mail about all builds, bothpassing and failing +@item failing +Only send mail about builds which fail +@item problem +Only send mail about a build which failed when the previous build has passed. +If your builds usually pass, then this will only send mail when a problem +occurs. +@end table + +@item builders +(list of strings). A list of builder names for which mail should be +sent. Defaults to None (send mail for all builds). Use either builders +or categories, but not both. + +@item categories +(list of strings). A list of category names to serve status +information for. Defaults to None (all categories). Use either +builders or categories, but not both. + +@item addLogs +(boolean). If True, include all build logs as attachments to the +messages. These can be quite large. This can also be set to a list of +log names, to send a subset of the logs. Defaults to False. + +@item addPatch +(boolean). If True, include the patch content if a patch was present. +Patches are usually used on a Try server. +Defaults to True. + +@item relayhost +(string). The host to which the outbound SMTP connection should be +made. Defaults to 'localhost' + +@item lookup +(implementor of @code{IEmailLookup}). Object which provides +IEmailLookup, which is responsible for mapping User names (which come +from the VC system) into valid email addresses. If not provided, the +notifier will only be able to send mail to the addresses in the +extraRecipients list. Most of the time you can use a simple Domain +instance. As a shortcut, you can pass as string: this will be treated +as if you had provided Domain(str). For example, +lookup='twistedmatrix.com' will allow mail to be sent to all +developers whose SVN usernames match their twistedmatrix.com account +names. See buildbot/status/mail.py for more details. + +@item messageFormatter + +This is a optional function that can be used to generate a custom mail message. +A @code{messageFormatter} function takes the mail mode (@code{mode}), builder +name (@code{name}), the build status (@code{build}), the result code +(@code{results}), and the BuildMaster status (@code{master_status}). It +returns a dictionary. The @code{body} key gives a string that is the complete +text of the message. The @code{type} key is the message type ('plain' or +'html'). The 'html' type should be used when generating an HTML message. The +@code{subject} key is optional, but gives the subject for the email. + +@item extraHeaders +(dictionary) A dictionary containing key/value pairs of extra headers to add +to sent e-mails. Both the keys and the values may be a WithProperties instance. + +@end table + +As a help to those writing @code{messageFormatter} functions, the following +table describes how to get some useful pieces of information from the various +status objects: + +@table @bullet +@item Name of the builder that generated this event +@code{name} +@item Name of the project +@code{master_status.getProjectName()} +@item MailNotifier mode +@code{mode} (one of @code{all, failing, problem, change, passing}) +@item Builder result as a string +@example +from buildbot.status.builder import Results +result_str = Results[results] +# one of 'success', 'warnings', 'failure', 'skipped', or 'exception' +@end example +@item URL to build page +@code{master_status.getURLForThing(build)} +@item URL to buildbot main page. +@code{master_status.getBuildbotURL()} +@item Build text +@code{build.getText()} +@item Mapping of property names to values +@code{build.getProperties()} (a @code{Properties} instance) +@item Slave name +@code{build.getSlavename()} +@item Build reason (from a forced build) +@code{build.getReason()} +@item List of responsible users +@code{build.getResponsibleUsers()} +@item Source information (only valid if ss is not None) +@example +ss = build.getSourceStamp() +if ss: + branch = ss.branch + revision = ss.revision + patch = ss.patch + changes = ss.changes # list +@end example +A change object has the following useful information: +@table @code +@item who +(str) who made this change +@item revision +(str) what VC revision is this change +@item branch +(str) on what branch did this change occur +@item when +(str) when did this change occur +@item files +(list of str) what files were affected in this change +@item comments +(str) comments reguarding the change. +@end table +The @code{Change} methods asText and asHTML return a list of strings with +the above information formatted. +@item Log information +@example +logs = list() +for log in build.getLogs(): + log_name = "%s.%s" % (log.getStep().getName(), log.getName()) + log_status, dummy = log.getStep().getResults() + log_body = log.getText().splitlines() # Note: can be VERY LARGE + log_url = '%s/steps/%s/logs/%s' % (master_status.getURLForThing(build), + log.getStep().getName(), + log.getName()) + logs.append((log_name, log_url, log_body, log_status)) +@end example +@end table + +@node IRC Bot +@section IRC Bot + +@cindex IRC +@stindex buildbot.status.words.IRC + + +The @code{buildbot.status.words.IRC} status target creates an IRC bot +which will attach to certain channels and be available for status +queries. It can also be asked to announce builds as they occur, or be +told to shut up. + +@example +from buildbot.status import words +irc = words.IRC("irc.example.org", "botnickname", + channels=["channel1", "channel2"], + password="mysecretpassword", + notify_events=@{ + 'exception': 1, + 'successToFailure': 1, + 'failureToSuccess': 1, + @}) +c['status'].append(irc) +@end example + +Take a look at the docstring for @code{words.IRC} for more details on +configuring this service. Note that the @code{useSSL} option requires +@code{PyOpenSSL}. The @code{password} argument, if provided, will be sent to +Nickserv to claim the nickname: some IRC servers will not allow clients to send +private messages until they have logged in with a password. + +To use the service, you address messages at the buildbot, either +normally (@code{botnickname: status}) or with private messages +(@code{/msg botnickname status}). The buildbot will respond in kind. + +Some of the commands currently available: + +@table @code + +@item list builders +Emit a list of all configured builders +@item status BUILDER +Announce the status of a specific Builder: what it is doing right now. +@item status all +Announce the status of all Builders +@item watch BUILDER +If the given Builder is currently running, wait until the Build is +finished and then announce the results. +@item last BUILDER +Return the results of the last build to run on the given Builder. +@item join CHANNEL +Join the given IRC channel +@item leave CHANNEL +Leave the given IRC channel +@item notify on|off|list EVENT +Report events relating to builds. If the command is issued as a +private message, then the report will be sent back as a private +message to the user who issued the command. Otherwise, the report +will be sent to the channel. Available events to be notified are: + +@table @code +@item started +A build has started +@item finished +A build has finished +@item success +A build finished successfully +@item failed +A build failed +@item exception +A build generated and exception +@item xToY +The previous build was x, but this one is Y, where x and Y are each +one of success, warnings, failure, exception (except Y is +capitalized). For example: successToFailure will notify if the +previous build was successful, but this one failed +@end table + +@item help COMMAND +Describe a command. Use @code{help commands} to get a list of known +commands. +@item source +Announce the URL of the Buildbot's home page. +@item version +Announce the version of this Buildbot. +@end table + +Additionally, the config file may specify default notification options +as shown in the example earlier. + +If the @code{allowForce=True} option was used, some addtional commands +will be available: + +@table @code +@item force build BUILDER REASON +Tell the given Builder to start a build of the latest code. The user +requesting the build and REASON are recorded in the Build status. The +buildbot will announce the build's status when it finishes. + +@item stop build BUILDER REASON +Terminate any running build in the given Builder. REASON will be added +to the build status to explain why it was stopped. You might use this +if you committed a bug, corrected it right away, and don't want to +wait for the first build (which is destined to fail) to complete +before starting the second (hopefully fixed) build. +@end table + +@node PBListener +@section PBListener + +@cindex PBListener +@stindex buildbot.status.client.PBListener + + +@example +import buildbot.status.client +pbl = buildbot.status.client.PBListener(port=int, user=str, + passwd=str) +c['status'].append(pbl) +@end example + +This sets up a PB listener on the given TCP port, to which a PB-based +status client can connect and retrieve status information. +@code{buildbot statusgui} (@pxref{statusgui}) is an example of such a +status client. The @code{port} argument can also be a strports +specification string. + +@node Writing New Status Plugins +@section Writing New Status Plugins + +TODO: this needs a lot more examples + +Each status plugin is an object which provides the +@code{twisted.application.service.IService} interface, which creates a +tree of Services with the buildmaster at the top [not strictly true]. +The status plugins are all children of an object which implements +@code{buildbot.interfaces.IStatus}, the main status object. From this +object, the plugin can retrieve anything it wants about current and +past builds. It can also subscribe to hear about new and upcoming +builds. + +Status plugins which only react to human queries (like the Waterfall +display) never need to subscribe to anything: they are idle until +someone asks a question, then wake up and extract the information they +need to answer it, then they go back to sleep. Plugins which need to +act spontaneously when builds complete (like the MailNotifier plugin) +need to subscribe to hear about new builds. + +If the status plugin needs to run network services (like the HTTP +server used by the Waterfall plugin), they can be attached as Service +children of the plugin itself, using the @code{IServiceCollection} +interface. + + + + diff --git a/docs/version.py b/docs/version.py new file mode 100644 index 00000000000..d08036597a8 --- /dev/null +++ b/docs/version.py @@ -0,0 +1,6 @@ +#! /usr/bin/env python + +gl = {} +execfile("../buildbot/__init__.py", gl) +version = gl['version'] +print "@set VERSION", version From fac0e586b6eecb46f90e2c74dad14f2ff4a5eb6d Mon Sep 17 00:00:00 2001 From: "Dustin J. Mitchell" Date: Sun, 14 Feb 2010 15:06:09 -0600 Subject: [PATCH 2/6] add pointer to API docs --- docs/buildbot.texinfo | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/docs/buildbot.texinfo b/docs/buildbot.texinfo index e8bb598ae5b..9992eae82af 100644 --- a/docs/buildbot.texinfo +++ b/docs/buildbot.texinfo @@ -109,6 +109,10 @@ are useful in @file{master.cfg}, the buildmaster's configuration file. Classes that are not listed here are generally internal things that admins are unlikely to have much use for. +A more complete collection of documentation from the source code itself is +available in the API documentation at +@uref{http://djmitche.github.com/buildbot/docs/}. + @heading Change Sources @printindex cs From 41b9ffa07b0d0663eca7a508c8fcf2cb4b57ee02 Mon Sep 17 00:00:00 2001 From: "Dustin J. Mitchell" Date: Sun, 14 Feb 2010 15:15:36 -0600 Subject: [PATCH 3/6] Support building docs in both split and full HTML --- .gitignore | 2 ++ docs/Makefile | 19 +++++++++++++------ 2 files changed, 15 insertions(+), 6 deletions(-) diff --git a/.gitignore b/.gitignore index 17016ae8b55..2692327aede 100644 --- a/.gitignore +++ b/.gitignore @@ -30,3 +30,5 @@ docs/images/status.png twisted/ docs/buildbot docs/version.texinfo +docs/docs.tgz +docs/latest diff --git a/docs/Makefile b/docs/Makefile index d4999e45ac2..7130b2b03b6 100644 --- a/docs/Makefile +++ b/docs/Makefile @@ -1,18 +1,25 @@ +all: docs.tgz + +VERSION=$(shell python -c 'g = {}; execfile("../buildbot/__init__.py",g); print g["version"]') buildbot.info: buildbot.texinfo - python version.py > version.texinfo + echo "@set VERSION $(VERSION)" > version.texinfo makeinfo --fill-column=70 $< buildbot.html: buildbot.texinfo images-png - python version.py > version.texinfo + echo "@set VERSION $(VERSION)" > version.texinfo makeinfo --no-split --html $< -buildbot: buildbot.texinfo images-png - python version.py > version.texinfo - makeinfo --html $< +docs.tgz: buildbot.html images-png + echo "@set VERSION $(VERSION)" > version.texinfo + rm -rf $(VERSION) + makeinfo --html --output $(VERSION) buildbot.texinfo + cp buildbot.html $(VERSION)/full.html + tar -zcf $@ $(VERSION) buildbot.ps: buildbot.texinfo images-eps - python version.py > version.texinfo + echo "@set VERSION $(VERSION)" > version.texinfo + rm -rf buildbot texi2dvi $< dvips buildbot.dvi rm buildbot.aux buildbot.cp buildbot.cps buildbot.fn buildbot.ky buildbot.log buildbot.pg buildbot.toc buildbot.tp buildbot.vr From c887205a20b33054a15786386a1eedc257c6c424 Mon Sep 17 00:00:00 2001 From: "Dustin J. Mitchell" Date: Sun, 14 Feb 2010 16:44:51 -0600 Subject: [PATCH 4/6] move 'Change Sources' under 'Configuration' --- docs/buildbot.texinfo | 18 +- ...rces.texinfo => cfg-changesources.texinfo} | 448 ++++++++---------- docs/concepts.texinfo | 6 +- docs/configuration.texinfo | 133 +++--- 4 files changed, 275 insertions(+), 330 deletions(-) rename docs/{changesources.texinfo => cfg-changesources.texinfo} (92%) diff --git a/docs/buildbot.texinfo b/docs/buildbot.texinfo index 9992eae82af..269884d2dc7 100644 --- a/docs/buildbot.texinfo +++ b/docs/buildbot.texinfo @@ -54,7 +54,6 @@ provided the copyright notice and this notice are preserved. * Installation:: Installing and setting up a Builbot system * Concepts:: What goes on in the buildbot's little mind. * Configuration:: Controlling the buildbot. -* Getting Source Code Changes:: Discovering when to run a build. * Build Process:: Controlling how each build is run. * Status Delivery:: Telling the world about the build's results. * Command-line Tool:: The 'buildbot' utility @@ -81,10 +80,6 @@ provided the copyright notice and this notice are preserved. @chapter Configuration @include configuration.texinfo -@node Getting Source Code Changes -@chapter Getting Source Code Changes -@include changesources.texinfo - @node Build Process @chapter Build Process @include buildprocess.texinfo @@ -131,18 +126,7 @@ available in the API documentation at @node Index of master.cfg keys @unnumbered Index of master.cfg keys -This is a list of all of the significant keys in @code{master.cfg}. Recall that -@code{master.cfg} is effectively a small python program with exactly one -responsibility: create a dictionary named @code{BuildmasterConfig}. The keys -of this dictionary are listed here. The beginning of the master.cfg file -typically starts with something like: - -@example -BuildmasterConfig = c = @{@} -@end example - -Therefore a config key of @code{change_source} will usually appear in -master.cfg as @code{c['change_source']}. +This is a list of all of the significant keys in @code{master.cfg}'s @code{BuildmasterConfig}. @printindex bc diff --git a/docs/changesources.texinfo b/docs/cfg-changesources.texinfo similarity index 92% rename from docs/changesources.texinfo rename to docs/cfg-changesources.texinfo index af842400d9c..3ce343d692c 100644 --- a/docs/changesources.texinfo +++ b/docs/cfg-changesources.texinfo @@ -1,12 +1,3 @@ -The most common way to use the Buildbot is centered around the idea of -@code{Source Trees}: a directory tree filled with source code of some form -which can be compiled and/or tested. Some projects use languages that don't -involve any compilation step: nevertheless there may be a @code{build} phase -where files are copied or rearranged into a form that is suitable for -installation. Some projects do not have unit tests, and the Buildbot is -merely helping to make sure that the sources can compile correctly. But in -all of these cases, the thing-being-tested is a single source tree. - A Version Control System mantains a source tree, and tells the buildmaster when it changes. The first step of each Build is typically to acquire a copy of some version of this tree. @@ -15,65 +6,33 @@ This chapter describes how the Buildbot learns about what Changes have occurred. For more information on VC systems and Changes, see @ref{Version Control Systems}. +Changes can be provided by a variety of ChangeSource types, although any given +project will typically have only a single ChangeSource active. This section +provides a description of all available ChangeSource types and explains how to +set up each of them. + +In general, each Buildmaster watches a single source tree. It is possible to +work around this, but true support for multi-tree builds remains elusive. @menu -* Change Sources:: -* Choosing ChangeSources:: +* Choosing a Change Source:: +* Configuring Change Sources:: * CVSToys - PBService:: * Mail-parsing ChangeSources:: * PBChangeSource:: * P4Source:: * BonsaiPoller:: * SVNPoller:: -* MercurialHook:: -* Bzr Hook:: * Bzr Poller:: @end menu - - -@node Change Sources -@section Change Sources - -@c TODO: rework this, the one-buildmaster-one-tree thing isn't quite -@c so narrow-minded anymore - -Each Buildmaster watches a single source tree. Changes can be provided -by a variety of ChangeSource types, however any given project will -typically have only a single ChangeSource active. This section -provides a description of all available ChangeSource types and -explains how to set up each of them. +@node Choosing a Change Source +@subsection Choosing a Change Source There are a variety of ChangeSources available, some of which are meant to be used in conjunction with other tools to deliver Change events from the VC repository to the buildmaster. -@itemize @bullet - -@item CVSToys -This ChangeSource opens a TCP connection from the buildmaster to a -waiting FreshCVS daemon that lives on the repository machine, and -subscribes to hear about Changes. - -@item MaildirSource -This one watches a local maildir-format inbox for email sent out by -the repository when a change is made. When a message arrives, it is -parsed to create the Change object. A variety of parsing functions are -available to accomodate different email-sending tools. - -@item PBChangeSource -This ChangeSource listens on a local TCP socket for inbound -connections from a separate tool. Usually, this tool would be run on -the VC repository machine in a commit hook. It is expected to connect -to the TCP socket and send a Change message over the network -connection. The @command{buildbot sendchange} command is one example -of a tool that knows how to send these messages, so you can write a -commit script for your VC system that calls it to deliver the Change. -There are other tools in the contrib/ directory that use the same -protocol. - -@end itemize - As a quick guide, here is a list of VC systems and the ChangeSources that might be useful with them. All of these ChangeSources are in the @code{buildbot.changes} module. @@ -145,9 +104,11 @@ All VC systems can be driven by a PBChangeSource and the If you write an email parsing function, they can also all be driven by a suitable @code{MaildirSource}. +@node Configuring Change Sources +@subsection Configuring Change Sources -@node Choosing ChangeSources -@section Choosing ChangeSources +@bcindex c['sources'] +@bcindex c['change_source'] The @code{master.cfg} configuration file has a dictionary key named @code{BuildmasterConfig['change_source']}, which holds the active @@ -158,13 +119,13 @@ key. Each buildmaster typically has just a single ChangeSource, since it is only watching a single source tree. But if, for some reason, you need multiple sources, just set @code{c['change_source']} to a list of -ChangeSources.. it will accept that too. +ChangeSources. @example s = FreshCVSSourceNewcred(host="host", port=4519, user="alice", passwd="secret", prefix="Twisted") -BuildmasterConfig['change_source'] = [s] +c['change_source'] = [s] @end example Each source tree has a nominal @code{top}. Each Change has a list of @@ -176,11 +137,11 @@ that @code{ChangeSource}. Files which are outside this sub-tree are ignored by the changesource: it does not generate Changes for those files. - @node CVSToys - PBService -@section CVSToys - PBService +@subsection CVSToys - PBService @csindex buildbot.changes.freshcvs.FreshCVSSource +@csindex buildbot.changes.freshcvs.FreshCVSSourceOldcred The @uref{http://purl.net/net/CVSToys, CVSToys} package provides a server which runs on the machine that hosts the CVS repository it @@ -190,14 +151,14 @@ changes. One of the notification options is named @code{PBService} and works by listening on a TCP port for clients. These clients subscribe to hear about commit notifications. -The buildmaster has a CVSToys-compatible @code{PBService} client built -in. There are two versions of it, one for old versions of CVSToys -(1.0.9 and earlier) which used the @code{oldcred} authentication -framework, and one for newer versions (1.0.10 and later) which use -@code{newcred}. Both are classes in the -@code{buildbot.changes.freshcvs} package. +The buildmaster has a CVSToys-compatible @code{PBService} client built in. +There are two versions of it, one for old versions of CVSToys (1.0.9 and +earlier, @code{FreshCVSSourceOldcred}) which used the @code{oldcred} +authentication framework, and one for newer versions (1.0.10 and later) which +use @code{newcred}. Both are classes in the @code{buildbot.changes.freshcvs} +package. -@code{FreshCVSSourceNewcred} objects are created with the following +@code{FreshCVSSource} objects are created with the following parameters: @table @samp @@ -236,9 +197,9 @@ using a username of 'foo' and a password of 'bar'. Then add a clause like this to your buildmaster's @file{master.cfg}: @example -BuildmasterConfig['change_source'] = FreshCVSSource("cvs.example.com", 4519, - "foo", "bar", - prefix="glib/") +c['change_source'] = FreshCVSSource("cvs.example.com", 4519, + "foo", "bar", + prefix="glib/") @end example where "cvs.example.com" is the host that is running the FreshCVS daemon, and @@ -251,7 +212,7 @@ it handles. @node Mail-parsing ChangeSources -@section Mail-parsing ChangeSources +@subsection Mail-parsing ChangeSources Many projects publish information about changes to their source tree by sending an email message out to a mailing list, frequently named @@ -266,8 +227,7 @@ buildmaster admin needs to arrange for these email messages to arrive in a place where the buildmaster can find them, and configure the buildmaster to parse the messages correctly. Once that is in place, the email parser will create Change objects and deliver them to the -Schedulers (see @pxref{Change Sources and Schedulers}) just -like any other ChangeSource. +Schedulers (see @pxref{Schedulers}) just like any other ChangeSource. There are two components to setting up an email-based ChangeSource. The first is to route the email messages to the buildmaster, which is @@ -291,10 +251,15 @@ c['change_source'] = SyncmailMaildirSource("~/maildir-buildbot", * Subscribing the Buildmaster:: * Using Maildirs:: * Parsing Email Change Messages:: +* FCMaildirSource:: +* SyncmailMaildirSource:: +* BonsaiMaildirSource:: +* SVNCommitEmailMaildirSource:: +* BzrLaunchpadEmailMaildirSource:: @end menu @node Subscribing the Buildmaster -@subsection Subscribing the Buildmaster +@subsubsection Subscribing the Buildmaster The recommended way to install the buildbot is to create a dedicated account for the buildmaster. If you do this, the account will probably @@ -321,7 +286,7 @@ Thus you can assign an extension address like @node Using Maildirs -@subsection Using Maildirs +@subsubsection Using Maildirs A ``maildir'' is a simple directory structure originally developed for qmail that allows safe atomic update without locking. Create a base @@ -353,7 +318,7 @@ this facility is not available, it polls the directory for new messages, every 10 seconds by default. @node Parsing Email Change Messages -@subsection Parsing Email Change Messages +@subsubsection Parsing Email Change Messages The second component to setting up an email-based ChangeSource is to parse the actual notices. This is highly dependent upon the VC system @@ -361,7 +326,7 @@ and commit script in use. A couple of common tools used to create these change emails are: -@table @samp +@table @b @item CVS @table @samp @@ -415,14 +380,6 @@ is ignored. If the filename @emph{does} start with the prefix, that prefix is stripped from the filename before any further processing is done. Thus the prefix usually ends with a slash. -@menu -* FCMaildirSource:: -* SyncmailMaildirSource:: -* BonsaiMaildirSource:: -* SVNCommitEmailMaildirSource:: -* BzrLaunchpadEmailMaildirSource:: -@end menu - @node FCMaildirSource @subsubsection FCMaildirSource @@ -532,7 +489,7 @@ c['change_source'] = BzrLaunchpadEmailMaildirSource("~/maildir-buildbot", branch @node PBChangeSource -@section PBChangeSource +@subsection PBChangeSource @csindex buildbot.changes.pb.PBChangeSource @@ -601,8 +558,169 @@ stripping prefixes at the sending end. @end table +The following hooks are useful for sending changes to a PBChangeSource: + +@menu +* Mercurial Hook:: +* Bzr Hook:: +@end menu + +@node Mercurial Hook +@subsubsection Mercurial Hook + +Since Mercurial is written in python, the hook script can invoke +Buildbot's @code{sendchange} function directly, rather than having to +spawn an external process. This function delivers the same sort of +changes as @code{buildbot sendchange} and the various hook scripts in +contrib/, so you'll need to add a @code{pb.PBChangeSource} to your +buildmaster to receive these changes. + +To set this up, first choose a Mercurial repository that represents +your central ``official'' source tree. This will be the same +repository that your buildslaves will eventually pull from. Install +Buildbot on the machine that hosts this repository, using the same +version of python as Mercurial is using (so that the Mercurial hook +can import code from buildbot). Then add the following to the +@code{.hg/hgrc} file in that repository, replacing the buildmaster +hostname/portnumber as appropriate for your buildbot: + +@example +[hooks] +changegroup.buildbot = python:buildbot.changes.hgbuildbot.hook + +[hgbuildbot] +master = buildmaster.example.org:9987 +@end example + +(Note that Mercurial lets you define multiple @code{changegroup} hooks +by giving them distinct names, like @code{changegroup.foo} and +@code{changegroup.bar}, which is why we use +@code{changegroup.buildbot} in this example. There is nothing magical +about the ``buildbot'' suffix in the hook name. The +@code{[hgbuildbot]} section @emph{is} special, however, as it is the +only section that the buildbot hook pays attention to.) + +Also note that this runs as a @code{changegroup} hook, rather than as +an @code{incoming} hook. The @code{changegroup} hook is run with +multiple revisions at a time (say, if multiple revisions are being +pushed to this repository in a single @command{hg push} command), +whereas the @code{incoming} hook is run with just one revision at a +time. The @code{hgbuildbot.hook} function will only work with the +@code{changegroup} hook. + +The @code{[hgbuildbot]} section has two other parameters that you +might specify, both of which control the name of the branch that is +attached to the changes coming from this hook. + +One common branch naming policy for Mercurial repositories is to use +it just like Darcs: each branch goes into a separate repository, and +all the branches for a single project share a common parent directory. +For example, you might have @file{/var/repos/PROJECT/trunk/} and +@file{/var/repos/PROJECT/release}. To use this style, use the +@code{branchtype = dirname} setting, which simply uses the last +component of the repository's enclosing directory as the branch name: + +@example +[hgbuildbot] +master = buildmaster.example.org:9987 +branchtype = dirname +@end example + +Another approach is to use Mercurial's built-in branches (the kind +created with @command{hg branch} and listed with @command{hg +branches}). This feature associates persistent names with particular +lines of descent within a single repository. (note that the buildbot +@code{source.Mercurial} checkout step does not yet support this kind +of branch). To have the commit hook deliver this sort of branch name +with the Change object, use @code{branchtype = inrepo}: + +@example +[hgbuildbot] +master = buildmaster.example.org:9987 +branchtype = inrepo +@end example + +Finally, if you want to simply specify the branchname directly, for +all changes, use @code{branch = BRANCHNAME}. This overrides +@code{branchtype}: + +@example +[hgbuildbot] +master = buildmaster.example.org:9987 +branch = trunk +@end example + +If you use @code{branch=} like this, you'll need to put a separate +.hgrc in each repository. If you use @code{branchtype=}, you may be +able to use the same .hgrc for all your repositories, stored in +@file{~/.hgrc} or @file{/etc/mercurial/hgrc}. + + +@node Bzr Hook +@subsubsection Bzr Hook + +Bzr is also written in Python, and the Bzr hook depends on Twisted to send the +changes. + +To install, put @code{contrib/bzr_buildbot.py} in one of your plugins +locations a bzr plugins directory (e.g., +@code{~/.bazaar/plugins}). Then, in one of your bazaar conf files (e.g., +@code{~/.bazaar/locations.conf}), set the location you want to connect with buildbot +with these keys: + +@table @code +@item buildbot_on +one of 'commit', 'push, or 'change'. Turns the plugin on to report changes via +commit, changes via push, or any changes to the trunk. 'change' is +recommended. + +@item buildbot_server +(required to send to a buildbot master) the URL of the buildbot master to +which you will connect (as of this writing, the same server and port to which +slaves connect). + +@item buildbot_port +(optional, defaults to 9989) the port of the buildbot master to which you will +connect (as of this writing, the same server and port to which slaves connect) + +@item buildbot_pqm +(optional, defaults to not pqm) Normally, the user that commits the revision +is the user that is responsible for the change. When run in a pqm (Patch Queue +Manager, see https://launchpad.net/pqm) environment, the user that commits is +the Patch Queue Manager, and the user that committed the *parent* revision is +responsible for the change. To turn on the pqm mode, set this value to any of +(case-insensitive) "Yes", "Y", "True", or "T". + +@item buildbot_dry_run +(optional, defaults to not a dry run) Normally, the post-commit hook will +attempt to communicate with the configured buildbot server and port. If this +parameter is included and any of (case-insensitive) "Yes", "Y", "True", or +"T", then the hook will simply print what it would have sent, but not attempt +to contact the buildbot master. + +@item buildbot_send_branch_name +(optional, defaults to not sending the branch name) If your buildbot's bzr +source build step uses a repourl, do *not* turn this on. If your buildbot's +bzr build step uses a baseURL, then you may set this value to any of +(case-insensitive) "Yes", "Y", "True", or "T" to have the buildbot master +append the branch name to the baseURL. + +@end table + +When buildbot no longer has a hardcoded password, it will be a configuration +option here as well. + +Here's a simple example that you might have in your +@code{~/.bazaar/locations.conf}. + +@example +[chroot-*:///var/local/myrepo/mybranch] +buildbot_on = change +buildbot_server = localhost +@end example + @node P4Source -@section P4Source +@subsection P4Source @csindex buildbot.changes.p4poller.P4Source @@ -658,7 +776,7 @@ c['change_source'] = s @end example @node BonsaiPoller -@section BonsaiPoller +@subsection BonsaiPoller @csindex buildbot.changes.bonsaipoller.BonsaiPoller @@ -675,7 +793,7 @@ arguments it accepts. @node SVNPoller -@section SVNPoller +@subsection SVNPoller @csindex buildbot.changes.svnpoller.SVNPoller @@ -953,162 +1071,8 @@ def my_file_splitter(path): return (branch, '/'.join(pieces)) @end example -@node MercurialHook -@section MercurialHook - -Since Mercurial is written in python, the hook script can invoke -Buildbot's @code{sendchange} function directly, rather than having to -spawn an external process. This function delivers the same sort of -changes as @code{buildbot sendchange} and the various hook scripts in -contrib/, so you'll need to add a @code{pb.PBChangeSource} to your -buildmaster to receive these changes. - -To set this up, first choose a Mercurial repository that represents -your central ``official'' source tree. This will be the same -repository that your buildslaves will eventually pull from. Install -Buildbot on the machine that hosts this repository, using the same -version of python as Mercurial is using (so that the Mercurial hook -can import code from buildbot). Then add the following to the -@code{.hg/hgrc} file in that repository, replacing the buildmaster -hostname/portnumber as appropriate for your buildbot: - -@example -[hooks] -changegroup.buildbot = python:buildbot.changes.hgbuildbot.hook - -[hgbuildbot] -master = buildmaster.example.org:9987 -@end example - -(Note that Mercurial lets you define multiple @code{changegroup} hooks -by giving them distinct names, like @code{changegroup.foo} and -@code{changegroup.bar}, which is why we use -@code{changegroup.buildbot} in this example. There is nothing magical -about the ``buildbot'' suffix in the hook name. The -@code{[hgbuildbot]} section @emph{is} special, however, as it is the -only section that the buildbot hook pays attention to.) - -Also note that this runs as a @code{changegroup} hook, rather than as -an @code{incoming} hook. The @code{changegroup} hook is run with -multiple revisions at a time (say, if multiple revisions are being -pushed to this repository in a single @command{hg push} command), -whereas the @code{incoming} hook is run with just one revision at a -time. The @code{hgbuildbot.hook} function will only work with the -@code{changegroup} hook. - -The @code{[hgbuildbot]} section has two other parameters that you -might specify, both of which control the name of the branch that is -attached to the changes coming from this hook. - -One common branch naming policy for Mercurial repositories is to use -it just like Darcs: each branch goes into a separate repository, and -all the branches for a single project share a common parent directory. -For example, you might have @file{/var/repos/PROJECT/trunk/} and -@file{/var/repos/PROJECT/release}. To use this style, use the -@code{branchtype = dirname} setting, which simply uses the last -component of the repository's enclosing directory as the branch name: - -@example -[hgbuildbot] -master = buildmaster.example.org:9987 -branchtype = dirname -@end example - -Another approach is to use Mercurial's built-in branches (the kind -created with @command{hg branch} and listed with @command{hg -branches}). This feature associates persistent names with particular -lines of descent within a single repository. (note that the buildbot -@code{source.Mercurial} checkout step does not yet support this kind -of branch). To have the commit hook deliver this sort of branch name -with the Change object, use @code{branchtype = inrepo}: - -@example -[hgbuildbot] -master = buildmaster.example.org:9987 -branchtype = inrepo -@end example - -Finally, if you want to simply specify the branchname directly, for -all changes, use @code{branch = BRANCHNAME}. This overrides -@code{branchtype}: - -@example -[hgbuildbot] -master = buildmaster.example.org:9987 -branch = trunk -@end example - -If you use @code{branch=} like this, you'll need to put a separate -.hgrc in each repository. If you use @code{branchtype=}, you may be -able to use the same .hgrc for all your repositories, stored in -@file{~/.hgrc} or @file{/etc/mercurial/hgrc}. - - -@node Bzr Hook -@section Bzr Hook - -Bzr is also written in Python, and the Bzr hook depends on Twisted to send the -changes. - -To install, put @code{contrib/bzr_buildbot.py} in one of your plugins -locations a bzr plugins directory (e.g., -@code{~/.bazaar/plugins}). Then, in one of your bazaar conf files (e.g., -@code{~/.bazaar/locations.conf}), set the location you want to connect with buildbot -with these keys: - -@table @code -@item buildbot_on -one of 'commit', 'push, or 'change'. Turns the plugin on to report changes via -commit, changes via push, or any changes to the trunk. 'change' is -recommended. - -@item buildbot_server -(required to send to a buildbot master) the URL of the buildbot master to -which you will connect (as of this writing, the same server and port to which -slaves connect). - -@item buildbot_port -(optional, defaults to 9989) the port of the buildbot master to which you will -connect (as of this writing, the same server and port to which slaves connect) - -@item buildbot_pqm -(optional, defaults to not pqm) Normally, the user that commits the revision -is the user that is responsible for the change. When run in a pqm (Patch Queue -Manager, see https://launchpad.net/pqm) environment, the user that commits is -the Patch Queue Manager, and the user that committed the *parent* revision is -responsible for the change. To turn on the pqm mode, set this value to any of -(case-insensitive) "Yes", "Y", "True", or "T". - -@item buildbot_dry_run -(optional, defaults to not a dry run) Normally, the post-commit hook will -attempt to communicate with the configured buildbot server and port. If this -parameter is included and any of (case-insensitive) "Yes", "Y", "True", or -"T", then the hook will simply print what it would have sent, but not attempt -to contact the buildbot master. - -@item buildbot_send_branch_name -(optional, defaults to not sending the branch name) If your buildbot's bzr -source build step uses a repourl, do *not* turn this on. If your buildbot's -bzr build step uses a baseURL, then you may set this value to any of -(case-insensitive) "Yes", "Y", "True", or "T" to have the buildbot master -append the branch name to the baseURL. - -@end table - -When buildbot no longer has a hardcoded password, it will be a configuration -option here as well. - -Here's a simple example that you might have in your -@code{~/.bazaar/locations.conf}. - -@example -[chroot-*:///var/local/myrepo/mybranch] -buildbot_on = change -buildbot_server = localhost -@end example - @node Bzr Poller -@section Bzr Poller +@subsection Bzr Poller If you cannot insert a Bzr hook in the server, you can use the Bzr Poller. To use, put @code{contrib/bzr_buildbot.py} somewhere that your buildbot diff --git a/docs/concepts.texinfo b/docs/concepts.texinfo index fbffaaf728c..1e4d0ff0ba4 100644 --- a/docs/concepts.texinfo +++ b/docs/concepts.texinfo @@ -4,7 +4,7 @@ configure it properly. @menu * Version Control Systems:: -* Schedulers:: +* Scheduling Builds:: * BuildSet:: * BuildRequest:: * Builder:: @@ -386,8 +386,8 @@ provides more detail for this Change, perhaps including the full file diffs. -@node Schedulers -@section Schedulers +@node Scheduling Builds +@section Scheduling Builds @cindex Scheduler diff --git a/docs/configuration.texinfo b/docs/configuration.texinfo index 8dd1386e0c6..49438a08fef 100644 --- a/docs/configuration.texinfo +++ b/docs/configuration.texinfo @@ -19,10 +19,13 @@ understand how to fill in each section properly. @menu * Config File Format:: +* Predefined Config File Symbols:: * Loading the Config File:: * Testing the Config File:: -* Defining the Project:: -* Change Sources and Schedulers:: +* Project Definitions:: +* Global Configuration:: +* Change Sources:: +* Schedulers:: * Merging BuildRequests:: * Prioritizing Builders:: * Setting the slaveport:: @@ -54,39 +57,39 @@ actions necessary to bring the buildbot up to date: Builders which are not changed are left untouched, and Builders which are modified get to keep their old event history. -Basic Python syntax: comments start with a hash character (``#''), -tuples are defined with @code{(parenthesis, pairs)}, arrays are -defined with @code{[square, brackets]}, tuples and arrays are mostly -interchangeable. Dictionaries (data structures which map ``keys'' to -``values'') are defined with curly braces: @code{@{'key1': 'value1', -'key2': 'value2'@} }. Function calls (and object instantiation) can use -named parameters, like @code{w = html.Waterfall(http_port=8010)}. - -The config file starts with a series of @code{import} statements, -which make various kinds of Steps and Status targets available for -later use. The main @code{BuildmasterConfig} dictionary is created, -then it is populated with a variety of keys. These keys are broken -roughly into the following sections, each of which is documented in -the rest of this chapter: +The beginning of the master.cfg file +typically starts with something like: -@itemize @bullet -@item -Project Definitions -@item -Change Sources / Schedulers -@item -Slaveport -@item -Buildslave Configuration -@item -Builders / Interlocks -@item -Status Targets -@item -Debug options -@end itemize +@example +BuildmasterConfig = c = @{@} +@end example + +Therefore a config key of @code{change_source} will usually appear in +master.cfg as @code{c['change_source']}. + +See @ref{Index of master.cfg keys} for a full list of @code{BuildMasterConfig} +keys. + +@heading Basic Python Syntax + +Python comments start with a hash character (``#''), tuples are defined with +@code{(parenthesis, pairs)}, and lists (arrays) are defined with @code{[square, +brackets]}. Tuples and listsk are mostly interchangeable. Dictionaries (data +structures which map ``keys'' to ``values'') are defined with curly braces: +@code{@{'key1': 'value1', 'key2': 'value2'@} }. Function calls (and object +instantiation) can use named parameters, like @code{w = +html.Waterfall(http_port=8010)}. + +The config file starts with a series of @code{import} statements, which make +various kinds of Steps and Status targets available for later use. The main +@code{BuildmasterConfig} dictionary is created, then it is populated with a +variety of keys, described section-by-section in subsequent chapters. + +@node Predefined Config File Symbols +@section Predefined Config File Symbols -The config file can use a few names which are placed into its namespace: +The following symbols are automatically available for use in the configuration +file. @table @code @item basedir @@ -94,10 +97,8 @@ the base directory for the buildmaster. This string has not been expanded, so it may start with a tilde. It needs to be expanded before use. The config file is located in @code{os.path.expanduser(os.path.join(basedir, 'master.cfg'))} - @end table - @node Loading the Config File @section Loading the Config File @@ -178,8 +179,8 @@ NameError: name 'bogus' is not defined @end example -@node Defining the Project -@section Defining the Project +@node Project Definitions +@section Project Definitions There are a couple of basic settings that you use to tell the buildbot what project it is working on. This information is used by status @@ -204,11 +205,10 @@ whole. HTML status displays will show @code{projectName} as a link to project's home page. @bcindex c['buildbotURL'] -The @code{buildbotURL} string should point to the location where the -buildbot's internal web server (usually the @code{html.Waterfall} -page) is visible. This typically uses the port number set when you -create the @code{Waterfall} object: the buildbot needs your help to -figure out a suitable externally-visible host name. +The @code{buildbotURL} string should point to the location where the buildbot's +internal web server is visible. This typically uses the port number set when +you create the @code{Waterfall} object: the buildbot needs your help to figure +out a suitable externally-visible host name. When status notices are sent to users (either by email or over IRC), @code{buildbotURL} will be used to create a URL to the specific build @@ -216,10 +216,22 @@ or problem that they are being notified about. It will also be made available to queriers (over IRC) who want to find out where to get more information about this buildbot. +@node Global Configuration +@section Global Configuration + +The keys in this section affect the operations of the buildmaster globally. + +@example +c['logCompressionLimit'] = 16384 +c['logCompressionMethod'] = 'gz' +c['logMaxSize'] = 1024*1024 # 1M +c['logMaxTailSize'] = 32768 +@end example + @bcindex c['logCompressionLimit'] The @code{logCompressionLimit} enables compression of build logs on disk for logs that are bigger than the given size, or disables that -completely if given @code{False}. The default value is 4k, which should +completely if set to @code{False}. The default value is 4k, which should be a reasonable default on most file systems. This setting has no impact on status plugins, and merely affects the required disk space on the master for build logs. @@ -244,37 +256,22 @@ contain the first @code{logMaxSize} bytes and the last @code{logMaxTailSize} bytes of output. Don't set this value too high, as the the tail of the log is kept in memory. -@node Change Sources and Schedulers -@section Change Sources and Schedulers - -@bcindex c['sources'] -@bcindex c['change_source'] - -The @code{c['change_source']} key is the ChangeSource -instance@footnote{To be precise, it is an object or a list of objects -which all implement the @code{buildbot.interfaces.IChangeSource} -Interface. It is unusual to have multiple ChangeSources, so this key -accepts either a single ChangeSource or a sequence of them.} that -defines how the buildmaster learns about source code changes. More -information about what goes here is available in @xref{Getting Source -Code Changes}. - -@example -from buildbot.changes.pb import PBChangeSource -c['change_source'] = PBChangeSource() -@end example - -(note: in buildbot-0.7.5 and earlier, this key was named -@code{c['sources']}, and required a list. @code{c['sources']} is -deprecated as of buildbot-0.7.6 and is scheduled to be removed in a -future release). - @bcindex c['changeHorizon'] The @code{c['changeHorizon']} key determines how many changes the master will keep a record of. One place these changes are displayed is on the waterfall page. This parameter defaults to 0, which means keep all changes indefinitely. +@c the remainder of this chapter is made up of other files, each of which acts +@c as if it was at the chapter level; the @lowersections/@raisesections make all +@c of this work out + +@node Change Sources +@section Change Sources +@include cfg-changesources.texinfo + +@node Schedulers +@section Schedulers @bcindex c['schedulers'] @code{c['schedulers']} is a list of Scheduler instances, each From b1f890eedf0afa3e3a4697fa1c4bce768f43e4fd Mon Sep 17 00:00:00 2001 From: "Dustin J. Mitchell" Date: Sun, 14 Feb 2010 16:50:26 -0600 Subject: [PATCH 5/6] break out 'Schedulers' section, add indices for each section --- docs/cfg-changesources.texinfo | 5 +- docs/cfg-global.texinfo | 44 +++ docs/cfg-projdef.texinfo | 35 +++ docs/cfg-schedulers.texinfo | 442 ++++++++++++++++++++++++++++ docs/configuration.texinfo | 522 +-------------------------------- 5 files changed, 528 insertions(+), 520 deletions(-) create mode 100644 docs/cfg-global.texinfo create mode 100644 docs/cfg-projdef.texinfo create mode 100644 docs/cfg-schedulers.texinfo diff --git a/docs/cfg-changesources.texinfo b/docs/cfg-changesources.texinfo index 3ce343d692c..7bfe2a488fc 100644 --- a/docs/cfg-changesources.texinfo +++ b/docs/cfg-changesources.texinfo @@ -24,6 +24,7 @@ work around this, but true support for multi-tree builds remains elusive. * BonsaiPoller:: * SVNPoller:: * Bzr Poller:: +* Change Source Index:: @end menu @node Choosing a Change Source @@ -1100,4 +1101,6 @@ responsible for the change. set this value to True if this is pointed against a PQM-managed branch. @end table - +@node Change Source Index +@subsection Change Source Index +@printindex cs diff --git a/docs/cfg-global.texinfo b/docs/cfg-global.texinfo new file mode 100644 index 00000000000..5c70a7af16a --- /dev/null +++ b/docs/cfg-global.texinfo @@ -0,0 +1,44 @@ +The keys in this section affect the operations of the buildmaster globally. + +@example +c['logCompressionLimit'] = 16384 +c['logCompressionMethod'] = 'gz' +c['logMaxSize'] = 1024*1024 # 1M +c['logMaxTailSize'] = 32768 +@end example + +@bcindex c['logCompressionLimit'] +The @code{logCompressionLimit} enables compression of build logs on +disk for logs that are bigger than the given size, or disables that +completely if set to @code{False}. The default value is 4k, which should +be a reasonable default on most file systems. This setting has no impact +on status plugins, and merely affects the required disk space on the +master for build logs. + +@bcindex c['logCompressionMethod'] +The @code{logCompressionMethod} controls what type of compression is used for +build logs. The default is 'bz2', the other valid option is 'gz'. 'bz2' +offers better compression at the expense of more CPU time. + +@bcindex c['logMaxSize'] +The @code{logMaxSize} parameter sets an upper limit (in bytes) to how large +logs from an individual build step can be. The default value is None, meaning +no upper limit to the log size. Any output exceeding @code{logMaxSize} will be +truncated, and a message to this effect will be added to the log's HEADER +channel. + +@bcindex c['logMaxTailSize'] +If @code{logMaxSize} is set, and the output from a step exceeds the maximum, +the @code{logMaxTailSize} parameter controls how much of the end of the build +log will be kept. The effect of setting this parameter is that the log will +contain the first @code{logMaxSize} bytes and the last @code{logMaxTailSize} +bytes of output. Don't set this value too high, as the the tail of the log is +kept in memory. + +@bcindex c['changeHorizon'] + +The @code{c['changeHorizon']} key determines how many changes the master will +keep a record of. One place these changes are displayed is on the waterfall +page. This parameter defaults to 0, which means keep all changes indefinitely. + + diff --git a/docs/cfg-projdef.texinfo b/docs/cfg-projdef.texinfo new file mode 100644 index 00000000000..5537f945f73 --- /dev/null +++ b/docs/cfg-projdef.texinfo @@ -0,0 +1,35 @@ +There are a couple of basic settings that you use to tell the buildbot +what project it is working on. This information is used by status +reporters to let users find out more about the codebase being +exercised by this particular Buildbot installation. + +@example +c['projectName'] = "Buildbot" +c['projectURL'] = "http://buildbot.sourceforge.net/" +c['buildbotURL'] = "http://localhost:8010/" +@end example + +@bcindex c['projectName'] +@code{projectName} is a short string will be used to describe the +project that this buildbot is working on. For example, it is used as +the title of the waterfall HTML page. + +@bcindex c['projectURL'] +@code{projectURL} is a string that gives a URL for the project as a +whole. HTML status displays will show @code{projectName} as a link to +@code{projectURL}, to provide a link from buildbot HTML pages to your +project's home page. + +@bcindex c['buildbotURL'] +The @code{buildbotURL} string should point to the location where the buildbot's +internal web server is visible. This typically uses the port number set when +you create the @code{Waterfall} object: the buildbot needs your help to figure +out a suitable externally-visible host name. + +When status notices are sent to users (either by email or over IRC), +@code{buildbotURL} will be used to create a URL to the specific build +or problem that they are being notified about. It will also be made +available to queriers (over IRC) who want to find out where to get +more information about this buildbot. + + diff --git a/docs/cfg-schedulers.texinfo b/docs/cfg-schedulers.texinfo new file mode 100644 index 00000000000..4922a50ea00 --- /dev/null +++ b/docs/cfg-schedulers.texinfo @@ -0,0 +1,442 @@ +@bcindex c['schedulers'] +@code{c['schedulers']} is a list of Scheduler instances, each +of which causes builds to be started on a particular set of +Builders. The two basic Scheduler classes you are likely to start +with are @code{Scheduler} and @code{Periodic}, but you can write a +customized subclass to implement more complicated build scheduling. + +Scheduler arguments +should always be specified by name (as keyword arguments), to allow +for future expansion: + +@example +sched = Scheduler(name="quick", builderNames=['lin', 'win']) +@end example + +All schedulers have several arguments in common: + +@table @code +@item name + +Each Scheduler must have a unique name. This is used in status +displays, and is also available in the build property @code{scheduler}. + +@item builderNames + +This is the set of builders which this scheduler should trigger, specified +as a list of names (strings). + +@item properties +@cindex Properties + +This is a dictionary specifying properties that will be transmitted +to all builds started by this scheduler. + +@end table + +Here is a brief catalog of the available Scheduler types. All these +Schedulers are classes in @code{buildbot.scheduler}, and the +docstrings there are the best source of documentation on the arguments +taken by each one. + +@menu +* Scheduler Scheduler:: +* AnyBranchScheduler:: +* Dependent Scheduler:: +* Periodic Scheduler:: +* Nightly Scheduler:: +* Try Schedulers:: +* Triggerable Scheduler:: +* Scheduler Index:: +@end menu + +@node Scheduler Scheduler +@subsection Scheduler Scheduler +@slindex buildbot.scheduler.Scheduler + +This is the original and still most popular Scheduler class. It follows +exactly one branch, and starts a configurable tree-stable-timer after +each change on that branch. When the timer expires, it starts a build +on some set of Builders. The Scheduler accepts a @code{fileIsImportant} +function which can be used to ignore some Changes if they do not +affect any ``important'' files. + +The arguments to this scheduler are: + +@table @code +@item name + +@item builderNames + +@item properties + +@item branch +This Scheduler will pay attention to a single branch, ignoring Changes +that occur on other branches. Setting @code{branch} equal to the +special value of @code{None} means it should only pay attention to +the default branch. Note that @code{None} is a keyword, not a string, +so you want to use @code{None} and not @code{"None"}. + +@item treeStableTimer +The Scheduler will wait for this many seconds before starting the +build. If new changes are made during this interval, the timer will be +restarted, so really the build will be started after a change and then +after this many seconds of inactivity. + +@item fileIsImportant +A callable which takes one argument, a Change instance, and returns +@code{True} if the change is worth building, and @code{False} if +it is not. Unimportant Changes are accumulated until the build is +triggered by an important change. The default value of None means +that all Changes are important. + +@item categories +A list of categories of changes that this scheduler will respond to. If this +is specified, then any non-matching changes are ignored. + +@end table + +Example: + +@example +from buildbot import scheduler +quick = scheduler.Scheduler(name="quick", + branch=None, + treeStableTimer=60, + builderNames=["quick-linux", "quick-netbsd"]) +full = scheduler.Scheduler(name="full", + branch=None, + treeStableTimer=5*60, + builderNames=["full-linux", "full-netbsd", "full-OSX"]) +c['schedulers'] = [quick, full] +@end example + +In this example, the two ``quick'' builders are triggered 60 seconds +after the tree has been changed. The ``full'' builds do not run quite +so quickly (they wait 5 minutes), so hopefully if the quick builds +fail due to a missing file or really simple typo, the developer can +discover and fix the problem before the full builds are started. Both +Schedulers only pay attention to the default branch: any changes +on other branches are ignored by these Schedulers. Each Scheduler +triggers a different set of Builders, referenced by name. + +@node AnyBranchScheduler +@subsection AnyBranchScheduler +@slindex buildbot.scheduler.AnyBranchScheduler + +This scheduler uses a tree-stable-timer like the default one, but +follows multiple branches at once. Each branch gets a separate timer. + +The arguments to this scheduler are: + +@table @code +@item name + +@item builderNames + +@item properties + +@item branches +This Scheduler will pay attention to any number of branches, ignoring +Changes that occur on other branches. Branches are specified just as +for the @code{Scheduler} class. + +@item treeStableTimer +The Scheduler will wait for this many seconds before starting the +build. If new changes are made during this interval, the timer will be +restarted, so really the build will be started after a change and then +after this many seconds of inactivity. + +@item fileIsImportant +A callable which takes one argument, a Change instance, and returns +@code{True} if the change is worth building, and @code{False} if +it is not. Unimportant Changes are accumulated until the build is +triggered by an important change. The default value of None means +that all Changes are important. + +@item categories +A list of categories of changes that this scheduler will respond to. If this +is specified, then any non-matching changes are ignored. + +@end table + +@node Dependent Scheduler +@subsection Dependent Scheduler +@cindex Dependent +@cindex Dependencies +@slindex buildbot.scheduler.Dependent + +It is common to wind up with one kind of build which should only be +performed if the same source code was successfully handled by some +other kind of build first. An example might be a packaging step: you +might only want to produce .deb or RPM packages from a tree that was +known to compile successfully and pass all unit tests. You could put +the packaging step in the same Build as the compile and testing steps, +but there might be other reasons to not do this (in particular you +might have several Builders worth of compiles/tests, but only wish to +do the packaging once). Another example is if you want to skip the +``full'' builds after a failing ``quick'' build of the same source +code. Or, if one Build creates a product (like a compiled library) +that is used by some other Builder, you'd want to make sure the +consuming Build is run @emph{after} the producing one. + +You can use ``Dependencies'' to express this relationship +to the Buildbot. There is a special kind of Scheduler named +@code{scheduler.Dependent} that will watch an ``upstream'' Scheduler +for builds to complete successfully (on all of its Builders). Each time +that happens, the same source code (i.e. the same @code{SourceStamp}) +will be used to start a new set of builds, on a different set of +Builders. This ``downstream'' scheduler doesn't pay attention to +Changes at all. It only pays attention to the upstream scheduler. + +If the build fails on any of the Builders in the upstream set, +the downstream builds will not fire. Note that, for SourceStamps +generated by a ChangeSource, the @code{revision} is None, meaning HEAD. +If any changes are committed between the time the upstream scheduler +begins its build and the time the dependent scheduler begins its +build, then those changes will be included in the downstream build. +See the @pxref{Triggerable Scheduler} for a more flexible dependency +mechanism that can avoid this problem. + +The keyword arguments to this scheduler are: + +@table @code +@item name + +@item builderNames + +@item properties + +@item upstream +The upstream scheduler to watch. Note that this is an ``instance'', +not the name of the scheduler. +@end table + +Example: + +@example +from buildbot import scheduler +tests = scheduler.Scheduler("just-tests", None, 5*60, + ["full-linux", "full-netbsd", "full-OSX"]) +package = scheduler.Dependent(name="build-package", + upstream=tests, # <- no quotes! + builderNames=["make-tarball", "make-deb", "make-rpm"]) +c['schedulers'] = [tests, package] +@end example + +@node Periodic Scheduler +@subsection Periodic Scheduler +@slindex buildbot.scheduler.Periodic + +This simple scheduler just triggers a build every N seconds. + +The arguments to this scheduler are: + +@table @code +@item name + +@item builderNames + +@item properties + +@item periodicBuildTimer +The time, in seconds, after which to start a build. +@end table + +Example: + +@example +from buildbot import scheduler +nightly = scheduler.Periodic(name="nightly", + builderNames=["full-solaris"], + periodicBuildTimer=24*60*60) +c['schedulers'] = [nightly] +@end example + +The Scheduler in this example just runs the full solaris build once +per day. Note that this Scheduler only lets you control the time +between builds, not the absolute time-of-day of each Build, so this +could easily wind up a ``daily'' or ``every afternoon'' scheduler +depending upon when it was first activated. + +@node Nightly Scheduler +@subsection Nightly Scheduler +@slindex buildbot.scheduler.Nightly + +This is highly configurable periodic build scheduler, which triggers +a build at particular times of day, week, month, or year. The +configuration syntax is very similar to the well-known @code{crontab} +format, in which you provide values for minute, hour, day, and month +(some of which can be wildcards), and a build is triggered whenever +the current time matches the given constraints. This can run a build +every night, every morning, every weekend, alternate Thursdays, +on your boss's birthday, etc. + +Pass some subset of @code{minute}, @code{hour}, @code{dayOfMonth}, +@code{month}, and @code{dayOfWeek}; each may be a single number or +a list of valid values. The builds will be triggered whenever the +current time matches these values. Wildcards are represented by a +'*' string. All fields default to a wildcard except 'minute', so +with no fields this defaults to a build every hour, on the hour. +The full list of parameters is: + +@table @code +@item name + +@item builderNames + +@item properties + +@item branch +The branch to build, just as for @code{Scheduler}. + +@item minute +The minute of the hour on which to start the build. This defaults +to 0, meaning an hourly build. + +@item hour +The hour of the day on which to start the build, in 24-hour notation. +This defaults to *, meaning every hour. + +@item month +The month in which to start the build, with January = 1. This defaults +to *, meaning every month. + +@item dayOfWeek +The day of the week to start a build, with Monday = 0. This defauls +to *, meaning every day of the week. + +@item onlyIfChanged +If this is true, then builds will not be scheduled at the designated time +unless the source has changed since the previous build. +@end table + +For example, the following master.cfg clause will cause a build to be +started every night at 3:00am: + +@example +s = scheduler.Nightly(name='nightly', + builderNames=['builder1', 'builder2'], + hour=3, + minute=0) +@end example + +This scheduler will perform a build each monday morning at 6:23am and +again at 8:23am, but only if someone has committed code in the interim: + +@example +s = scheduler.Nightly(name='BeforeWork', + builderNames=['builder1'], + dayOfWeek=0, + hour=[6,8], + minute=23, + onlyIfChanged=True) +@end example + +The following runs a build every two hours, using Python's @code{range} +function: + +@example +s = Nightly(name='every2hours', + builderNames=['builder1'], + hour=range(0, 24, 2)) +@end example + +Finally, this example will run only on December 24th: + +@example +s = Nightly(name='SleighPreflightCheck', + builderNames=['flying_circuits', 'radar'], + month=12, + dayOfMonth=24, + hour=12, + minute=0) +@end example + +@node Try Schedulers +@subsection Try Schedulers +@slindex buildbot.scheduler.Try_Jobdir +@slindex buildbot.scheduler.Try_Userpass + +This scheduler allows developers to use the @code{buildbot try} +command to trigger builds of code they have not yet committed. See +@ref{try} for complete details. + +Two implementations are available: @code{Try_Jobdir} and +@code{Try_Userpass}. The former monitors a job directory, specified +by the @code{jobdir} parameter, while the latter listens for PB +connections on a specific @code{port}, and authenticates against +@code{userport}. + +@node Triggerable Scheduler +@subsection Triggerable Scheduler +@cindex Triggers +@slindex buildbot.scheduler.Triggerable + +The @code{Triggerable} scheduler waits to be triggered by a Trigger +step (see @ref{Triggering Schedulers}) in another build. That step +can optionally wait for the scheduler's builds to complete. This +provides two advantages over Dependent schedulers. First, the same +scheduler can be triggered from multiple builds. Second, the ability +to wait for a Triggerable's builds to complete provides a form of +"subroutine call", where one or more builds can "call" a scheduler +to perform some work for them, perhaps on other buildslaves. + +The parameters are just the basics: + +@table @code +@item name +@item builderNames +@item properties +@end table + +This class is only useful in conjunction with the @code{Trigger} step. +Here is a fully-worked example: + +@example +from buildbot import scheduler +from buildbot.process import factory +from buildbot.steps import trigger + +checkin = scheduler.Scheduler(name="checkin", + branch=None, + treeStableTimer=5*60, + builderNames=["checkin"]) +nightly = scheduler.Nightly(name='nightly', + builderNames=['nightly'], + hour=3, + minute=0) + +mktarball = scheduler.Triggerable(name="mktarball", + builderNames=["mktarball"]) +build = scheduler.Triggerable(name="build-all-platforms", + builderNames=["build-all-platforms"]) +test = scheduler.Triggerable(name="distributed-test", + builderNames=["distributed-test"]) +package = scheduler.Triggerable(name="package-all-platforms", + builderNames=["package-all-platforms"]) + +c['schedulers'] = [mktarball, checkin, nightly, build, test, package] + +# on checkin, make a tarball, build it, and test it +checkin_factory = factory.BuildFactory() +checkin_factory.addStep(trigger.Trigger(schedulerNames=['mktarball'], + waitForFinish=True)) +checkin_factory.addStep(trigger.Trigger(schedulerNames=['build-all-platforms'], + waitForFinish=True)) +checkin_factory.addStep(trigger.Trigger(schedulerNames=['distributed-test'], + waitForFinish=True)) + +# and every night, make a tarball, build it, and package it +nightly_factory = factory.BuildFactory() +nightly_factory.addStep(trigger.Trigger(schedulerNames=['mktarball'], + waitForFinish=True)) +nightly_factory.addStep(trigger.Trigger(schedulerNames=['build-all-platforms'], + waitForFinish=True)) +nightly_factory.addStep(trigger.Trigger(schedulerNames=['package-all-platforms'], + waitForFinish=True)) +@end example + +@node Scheduler Index +@subsection Scheduler Index +@printindex sl diff --git a/docs/configuration.texinfo b/docs/configuration.texinfo index 49438a08fef..31cef0568d9 100644 --- a/docs/configuration.texinfo +++ b/docs/configuration.texinfo @@ -181,90 +181,11 @@ NameError: name 'bogus' is not defined @node Project Definitions @section Project Definitions - -There are a couple of basic settings that you use to tell the buildbot -what project it is working on. This information is used by status -reporters to let users find out more about the codebase being -exercised by this particular Buildbot installation. - -@example -c['projectName'] = "Buildbot" -c['projectURL'] = "http://buildbot.sourceforge.net/" -c['buildbotURL'] = "http://localhost:8010/" -@end example - -@bcindex c['projectName'] -@code{projectName} is a short string will be used to describe the -project that this buildbot is working on. For example, it is used as -the title of the waterfall HTML page. - -@bcindex c['projectURL'] -@code{projectURL} is a string that gives a URL for the project as a -whole. HTML status displays will show @code{projectName} as a link to -@code{projectURL}, to provide a link from buildbot HTML pages to your -project's home page. - -@bcindex c['buildbotURL'] -The @code{buildbotURL} string should point to the location where the buildbot's -internal web server is visible. This typically uses the port number set when -you create the @code{Waterfall} object: the buildbot needs your help to figure -out a suitable externally-visible host name. - -When status notices are sent to users (either by email or over IRC), -@code{buildbotURL} will be used to create a URL to the specific build -or problem that they are being notified about. It will also be made -available to queriers (over IRC) who want to find out where to get -more information about this buildbot. +@include cfg-projdef.texinfo @node Global Configuration @section Global Configuration - -The keys in this section affect the operations of the buildmaster globally. - -@example -c['logCompressionLimit'] = 16384 -c['logCompressionMethod'] = 'gz' -c['logMaxSize'] = 1024*1024 # 1M -c['logMaxTailSize'] = 32768 -@end example - -@bcindex c['logCompressionLimit'] -The @code{logCompressionLimit} enables compression of build logs on -disk for logs that are bigger than the given size, or disables that -completely if set to @code{False}. The default value is 4k, which should -be a reasonable default on most file systems. This setting has no impact -on status plugins, and merely affects the required disk space on the -master for build logs. - -@bcindex c['logCompressionMethod'] -The @code{logCompressionMethod} controls what type of compression is used for -build logs. The default is 'bz2', the other valid option is 'gz'. 'bz2' -offers better compression at the expense of more CPU time. - -@bcindex c['logMaxSize'] -The @code{logMaxSize} parameter sets an upper limit (in bytes) to how large -logs from an individual build step can be. The default value is None, meaning -no upper limit to the log size. Any output exceeding @code{logMaxSize} will be -truncated, and a message to this effect will be added to the log's HEADER -channel. - -@bcindex c['logMaxTailSize'] -If @code{logMaxSize} is set, and the output from a step exceeds the maximum, -the @code{logMaxTailSize} parameter controls how much of the end of the build -log will be kept. The effect of setting this parameter is that the log will -contain the first @code{logMaxSize} bytes and the last @code{logMaxTailSize} -bytes of output. Don't set this value too high, as the the tail of the log is -kept in memory. - -@bcindex c['changeHorizon'] - -The @code{c['changeHorizon']} key determines how many changes the master will -keep a record of. One place these changes are displayed is on the waterfall -page. This parameter defaults to 0, which means keep all changes indefinitely. - -@c the remainder of this chapter is made up of other files, each of which acts -@c as if it was at the chapter level; the @lowersections/@raisesections make all -@c of this work out +@include cfg-global.texinfo @node Change Sources @section Change Sources @@ -272,444 +193,7 @@ page. This parameter defaults to 0, which means keep all changes indefinitely. @node Schedulers @section Schedulers - -@bcindex c['schedulers'] -@code{c['schedulers']} is a list of Scheduler instances, each -of which causes builds to be started on a particular set of -Builders. The two basic Scheduler classes you are likely to start -with are @code{Scheduler} and @code{Periodic}, but you can write a -customized subclass to implement more complicated build scheduling. - -Scheduler arguments -should always be specified by name (as keyword arguments), to allow -for future expansion: - -@example -sched = Scheduler(name="quick", builderNames=['lin', 'win']) -@end example - -All schedulers have several arguments in common: - -@table @code -@item name - -Each Scheduler must have a unique name. This is used in status -displays, and is also available in the build property @code{scheduler}. - -@item builderNames - -This is the set of builders which this scheduler should trigger, specified -as a list of names (strings). - -@item properties -@cindex Properties - -This is a dictionary specifying properties that will be transmitted -to all builds started by this scheduler. - -@end table - -Here is a brief catalog of the available Scheduler types. All these -Schedulers are classes in @code{buildbot.scheduler}, and the -docstrings there are the best source of documentation on the arguments -taken by each one. - -@menu -* Scheduler Scheduler:: -* AnyBranchScheduler:: -* Dependent Scheduler:: -* Periodic Scheduler:: -* Nightly Scheduler:: -* Try Schedulers:: -* Triggerable Scheduler:: -@end menu - -@node Scheduler Scheduler -@subsection Scheduler Scheduler -@slindex buildbot.scheduler.Scheduler - -This is the original and still most popular Scheduler class. It follows -exactly one branch, and starts a configurable tree-stable-timer after -each change on that branch. When the timer expires, it starts a build -on some set of Builders. The Scheduler accepts a @code{fileIsImportant} -function which can be used to ignore some Changes if they do not -affect any ``important'' files. - -The arguments to this scheduler are: - -@table @code -@item name - -@item builderNames - -@item properties - -@item branch -This Scheduler will pay attention to a single branch, ignoring Changes -that occur on other branches. Setting @code{branch} equal to the -special value of @code{None} means it should only pay attention to -the default branch. Note that @code{None} is a keyword, not a string, -so you want to use @code{None} and not @code{"None"}. - -@item treeStableTimer -The Scheduler will wait for this many seconds before starting the -build. If new changes are made during this interval, the timer will be -restarted, so really the build will be started after a change and then -after this many seconds of inactivity. - -@item fileIsImportant -A callable which takes one argument, a Change instance, and returns -@code{True} if the change is worth building, and @code{False} if -it is not. Unimportant Changes are accumulated until the build is -triggered by an important change. The default value of None means -that all Changes are important. - -@item categories -A list of categories of changes that this scheduler will respond to. If this -is specified, then any non-matching changes are ignored. - -@end table - -Example: - -@example -from buildbot import scheduler -quick = scheduler.Scheduler(name="quick", - branch=None, - treeStableTimer=60, - builderNames=["quick-linux", "quick-netbsd"]) -full = scheduler.Scheduler(name="full", - branch=None, - treeStableTimer=5*60, - builderNames=["full-linux", "full-netbsd", "full-OSX"]) -c['schedulers'] = [quick, full] -@end example - -In this example, the two ``quick'' builders are triggered 60 seconds -after the tree has been changed. The ``full'' builds do not run quite -so quickly (they wait 5 minutes), so hopefully if the quick builds -fail due to a missing file or really simple typo, the developer can -discover and fix the problem before the full builds are started. Both -Schedulers only pay attention to the default branch: any changes -on other branches are ignored by these Schedulers. Each Scheduler -triggers a different set of Builders, referenced by name. - -@node AnyBranchScheduler -@subsection AnyBranchScheduler -@slindex buildbot.scheduler.AnyBranchScheduler - -This scheduler uses a tree-stable-timer like the default one, but -follows multiple branches at once. Each branch gets a separate timer. - -The arguments to this scheduler are: - -@table @code -@item name - -@item builderNames - -@item properties - -@item branches -This Scheduler will pay attention to any number of branches, ignoring -Changes that occur on other branches. Branches are specified just as -for the @code{Scheduler} class. - -@item treeStableTimer -The Scheduler will wait for this many seconds before starting the -build. If new changes are made during this interval, the timer will be -restarted, so really the build will be started after a change and then -after this many seconds of inactivity. - -@item fileIsImportant -A callable which takes one argument, a Change instance, and returns -@code{True} if the change is worth building, and @code{False} if -it is not. Unimportant Changes are accumulated until the build is -triggered by an important change. The default value of None means -that all Changes are important. - -@item categories -A list of categories of changes that this scheduler will respond to. If this -is specified, then any non-matching changes are ignored. - -@end table - -@node Dependent Scheduler -@subsection Dependent Scheduler -@cindex Dependent -@cindex Dependencies -@slindex buildbot.scheduler.Dependent - -It is common to wind up with one kind of build which should only be -performed if the same source code was successfully handled by some -other kind of build first. An example might be a packaging step: you -might only want to produce .deb or RPM packages from a tree that was -known to compile successfully and pass all unit tests. You could put -the packaging step in the same Build as the compile and testing steps, -but there might be other reasons to not do this (in particular you -might have several Builders worth of compiles/tests, but only wish to -do the packaging once). Another example is if you want to skip the -``full'' builds after a failing ``quick'' build of the same source -code. Or, if one Build creates a product (like a compiled library) -that is used by some other Builder, you'd want to make sure the -consuming Build is run @emph{after} the producing one. - -You can use ``Dependencies'' to express this relationship -to the Buildbot. There is a special kind of Scheduler named -@code{scheduler.Dependent} that will watch an ``upstream'' Scheduler -for builds to complete successfully (on all of its Builders). Each time -that happens, the same source code (i.e. the same @code{SourceStamp}) -will be used to start a new set of builds, on a different set of -Builders. This ``downstream'' scheduler doesn't pay attention to -Changes at all. It only pays attention to the upstream scheduler. - -If the build fails on any of the Builders in the upstream set, -the downstream builds will not fire. Note that, for SourceStamps -generated by a ChangeSource, the @code{revision} is None, meaning HEAD. -If any changes are committed between the time the upstream scheduler -begins its build and the time the dependent scheduler begins its -build, then those changes will be included in the downstream build. -See the @pxref{Triggerable Scheduler} for a more flexible dependency -mechanism that can avoid this problem. - -The keyword arguments to this scheduler are: - -@table @code -@item name - -@item builderNames - -@item properties - -@item upstream -The upstream scheduler to watch. Note that this is an ``instance'', -not the name of the scheduler. -@end table - -Example: - -@example -from buildbot import scheduler -tests = scheduler.Scheduler("just-tests", None, 5*60, - ["full-linux", "full-netbsd", "full-OSX"]) -package = scheduler.Dependent(name="build-package", - upstream=tests, # <- no quotes! - builderNames=["make-tarball", "make-deb", "make-rpm"]) -c['schedulers'] = [tests, package] -@end example - -@node Periodic Scheduler -@subsection Periodic Scheduler -@slindex buildbot.scheduler.Periodic - -This simple scheduler just triggers a build every N seconds. - -The arguments to this scheduler are: - -@table @code -@item name - -@item builderNames - -@item properties - -@item periodicBuildTimer -The time, in seconds, after which to start a build. -@end table - -Example: - -@example -from buildbot import scheduler -nightly = scheduler.Periodic(name="nightly", - builderNames=["full-solaris"], - periodicBuildTimer=24*60*60) -c['schedulers'] = [nightly] -@end example - -The Scheduler in this example just runs the full solaris build once -per day. Note that this Scheduler only lets you control the time -between builds, not the absolute time-of-day of each Build, so this -could easily wind up a ``daily'' or ``every afternoon'' scheduler -depending upon when it was first activated. - -@node Nightly Scheduler -@subsection Nightly Scheduler -@slindex buildbot.scheduler.Nightly - -This is highly configurable periodic build scheduler, which triggers -a build at particular times of day, week, month, or year. The -configuration syntax is very similar to the well-known @code{crontab} -format, in which you provide values for minute, hour, day, and month -(some of which can be wildcards), and a build is triggered whenever -the current time matches the given constraints. This can run a build -every night, every morning, every weekend, alternate Thursdays, -on your boss's birthday, etc. - -Pass some subset of @code{minute}, @code{hour}, @code{dayOfMonth}, -@code{month}, and @code{dayOfWeek}; each may be a single number or -a list of valid values. The builds will be triggered whenever the -current time matches these values. Wildcards are represented by a -'*' string. All fields default to a wildcard except 'minute', so -with no fields this defaults to a build every hour, on the hour. -The full list of parameters is: - -@table @code -@item name - -@item builderNames - -@item properties - -@item branch -The branch to build, just as for @code{Scheduler}. - -@item minute -The minute of the hour on which to start the build. This defaults -to 0, meaning an hourly build. - -@item hour -The hour of the day on which to start the build, in 24-hour notation. -This defaults to *, meaning every hour. - -@item month -The month in which to start the build, with January = 1. This defaults -to *, meaning every month. - -@item dayOfWeek -The day of the week to start a build, with Monday = 0. This defauls -to *, meaning every day of the week. - -@item onlyIfChanged -If this is true, then builds will not be scheduled at the designated time -unless the source has changed since the previous build. -@end table - -For example, the following master.cfg clause will cause a build to be -started every night at 3:00am: - -@example -s = scheduler.Nightly(name='nightly', - builderNames=['builder1', 'builder2'], - hour=3, - minute=0) -@end example - -This scheduler will perform a build each monday morning at 6:23am and -again at 8:23am, but only if someone has committed code in the interim: - -@example -s = scheduler.Nightly(name='BeforeWork', - builderNames=['builder1'], - dayOfWeek=0, - hour=[6,8], - minute=23, - onlyIfChanged=True) -@end example - -The following runs a build every two hours, using Python's @code{range} -function: - -@example -s = Nightly(name='every2hours', - builderNames=['builder1'], - hour=range(0, 24, 2)) -@end example - -Finally, this example will run only on December 24th: - -@example -s = Nightly(name='SleighPreflightCheck', - builderNames=['flying_circuits', 'radar'], - month=12, - dayOfMonth=24, - hour=12, - minute=0) -@end example - -@node Try Schedulers -@subsection Try Schedulers -@slindex buildbot.scheduler.Try_Jobdir -@slindex buildbot.scheduler.Try_Userpass - -This scheduler allows developers to use the @code{buildbot try} -command to trigger builds of code they have not yet committed. See -@ref{try} for complete details. - -Two implementations are available: @code{Try_Jobdir} and -@code{Try_Userpass}. The former monitors a job directory, specified -by the @code{jobdir} parameter, while the latter listens for PB -connections on a specific @code{port}, and authenticates against -@code{userport}. - -@node Triggerable Scheduler -@subsection Triggerable Scheduler -@cindex Triggers -@slindex buildbot.scheduler.Triggerable - -The @code{Triggerable} scheduler waits to be triggered by a Trigger -step (see @ref{Triggering Schedulers}) in another build. That step -can optionally wait for the scheduler's builds to complete. This -provides two advantages over Dependent schedulers. First, the same -scheduler can be triggered from multiple builds. Second, the ability -to wait for a Triggerable's builds to complete provides a form of -"subroutine call", where one or more builds can "call" a scheduler -to perform some work for them, perhaps on other buildslaves. - -The parameters are just the basics: - -@table @code -@item name -@item builderNames -@item properties -@end table - -This class is only useful in conjunction with the @code{Trigger} step. -Here is a fully-worked example: - -@example -from buildbot import scheduler -from buildbot.process import factory -from buildbot.steps import trigger - -checkin = scheduler.Scheduler(name="checkin", - branch=None, - treeStableTimer=5*60, - builderNames=["checkin"]) -nightly = scheduler.Nightly(name='nightly', - builderNames=['nightly'], - hour=3, - minute=0) - -mktarball = scheduler.Triggerable(name="mktarball", - builderNames=["mktarball"]) -build = scheduler.Triggerable(name="build-all-platforms", - builderNames=["build-all-platforms"]) -test = scheduler.Triggerable(name="distributed-test", - builderNames=["distributed-test"]) -package = scheduler.Triggerable(name="package-all-platforms", - builderNames=["package-all-platforms"]) - -c['schedulers'] = [mktarball, checkin, nightly, build, test, package] - -# on checkin, make a tarball, build it, and test it -checkin_factory = factory.BuildFactory() -checkin_factory.addStep(trigger.Trigger(schedulerNames=['mktarball'], - waitForFinish=True)) -checkin_factory.addStep(trigger.Trigger(schedulerNames=['build-all-platforms'], - waitForFinish=True)) -checkin_factory.addStep(trigger.Trigger(schedulerNames=['distributed-test'], - waitForFinish=True)) - -# and every night, make a tarball, build it, and package it -nightly_factory = factory.BuildFactory() -nightly_factory.addStep(trigger.Trigger(schedulerNames=['mktarball'], - waitForFinish=True)) -nightly_factory.addStep(trigger.Trigger(schedulerNames=['build-all-platforms'], - waitForFinish=True)) -nightly_factory.addStep(trigger.Trigger(schedulerNames=['package-all-platforms'], - waitForFinish=True)) -@end example +@include cfg-schedulers.texinfo @node Merging BuildRequests @section Merging BuildRequests From abb32cfff604870a4ea70b3bf87e50418ffdcc52 Mon Sep 17 00:00:00 2001 From: "Dustin J. Mitchell" Date: Mon, 15 Feb 2010 09:42:23 -0600 Subject: [PATCH 6/6] WIP - look for comment to see where I was --- docs/cfg-global.texinfo | 116 ++++++++++++++++++++++++++++++++++++- docs/cfg-projdef.texinfo | 35 ----------- docs/configuration.texinfo | 69 +--------------------- 3 files changed, 116 insertions(+), 104 deletions(-) delete mode 100644 docs/cfg-projdef.texinfo diff --git a/docs/cfg-global.texinfo b/docs/cfg-global.texinfo index 5c70a7af16a..06799fcf2f5 100644 --- a/docs/cfg-global.texinfo +++ b/docs/cfg-global.texinfo @@ -1,5 +1,53 @@ The keys in this section affect the operations of the buildmaster globally. +@menu +* Project Definitions:: +* Log Handling:: +* Data Lifetime:: +* Merging BuildRequests:: +* Prioritizing Builders:: +@end menu + +@node Project Definitions +@subsection Project Definitions + +There are a couple of basic settings that you use to tell the buildbot +what project it is working on. This information is used by status +reporters to let users find out more about the codebase being +exercised by this particular Buildbot installation. + +@example +c['projectName'] = "Buildbot" +c['projectURL'] = "http://buildbot.sourceforge.net/" +c['buildbotURL'] = "http://localhost:8010/" +@end example + +@bcindex c['projectName'] +@code{projectName} is a short string will be used to describe the +project that this buildbot is working on. For example, it is used as +the title of the waterfall HTML page. + +@bcindex c['projectURL'] +@code{projectURL} is a string that gives a URL for the project as a +whole. HTML status displays will show @code{projectName} as a link to +@code{projectURL}, to provide a link from buildbot HTML pages to your +project's home page. + +@bcindex c['buildbotURL'] +The @code{buildbotURL} string should point to the location where the buildbot's +internal web server is visible. This typically uses the port number set when +you create the @code{Waterfall} object: the buildbot needs your help to figure +out a suitable externally-visible host name. + +When status notices are sent to users (either by email or over IRC), +@code{buildbotURL} will be used to create a URL to the specific build +or problem that they are being notified about. It will also be made +available to queriers (over IRC) who want to find out where to get +more information about this buildbot. + +@node Log Handling +@subsection Log Handling + @example c['logCompressionLimit'] = 16384 c['logCompressionMethod'] = 'gz' @@ -35,10 +83,76 @@ contain the first @code{logMaxSize} bytes and the last @code{logMaxTailSize} bytes of output. Don't set this value too high, as the the tail of the log is kept in memory. -@bcindex c['changeHorizon'] +@node Data Lifetime +@subsection Data Lifetime +@example +c['changeHorizon'] = 200 +@end example + +@bcindex c['changeHorizon'] The @code{c['changeHorizon']} key determines how many changes the master will keep a record of. One place these changes are displayed is on the waterfall page. This parameter defaults to 0, which means keep all changes indefinitely. +@node Merging BuildRequests +@subsection Merging BuildRequests + +@bcindex c['mergeRequests'] + +By default, buildbot merges BuildRequests that have the compatible +SourceStamps. This behaviour can be customized with the +@code{c['mergeRequests']} configuration key. This key specifies a function +which is called with three arguments: a @code{Builder} and two +@code{BuildRequest} objects. It should return true if the requests can be +merged. For example: + +@example +def mergeRequests(builder, req1, req2): + """Don't merge buildrequest at all""" + return False +c['mergeRequests'] = mergeRequests +@end example + +In many cases, the details of the SourceStamps and BuildRequests are important. +In this example, only BuildRequests with the same "reason" are merged; thus +developers forcing builds for different reasons will see distinct builds. + +@example +def mergeRequests(builder, req1, req2): + if req1.source.canBeMergedWith(req2.source) and req1.reason == req2.reason: + return True + return False +c['mergeRequests'] = mergeRequests +@end example + +@node Prioritizing Builders +@subsection Prioritizing Builders + +@bcindex c['prioritizeBuilders'] + +By default, buildbot will attempt to start builds on builders in order from the +builder with the oldest pending request to the newest. This behaviour can be +customized with the @code{c['prioritizeBuilders']} configuration key. +This key specifies a function which is called with two arguments: a +@code{BuildMaster} and a list of @code{Builder} objects. It +should return a list of @code{Builder} objects in the desired order. +It may also remove items from the list if builds should not be started +on those builders. + +@example +def prioritizeBuilders(buildmaster, builders): + """Prioritize builders. 'finalRelease' builds have the highest + priority, so they should be built before running tests, or + creating builds.""" + builderPriorities = @{ + "finalRelease": 0, + "test": 1, + "build": 2, + @} + builders.sort(key=lambda b: builderPriorities.get(b.name, 0)) + return builders + +c['prioritizeBuilders'] = prioritizeBuilders +@end example diff --git a/docs/cfg-projdef.texinfo b/docs/cfg-projdef.texinfo deleted file mode 100644 index 5537f945f73..00000000000 --- a/docs/cfg-projdef.texinfo +++ /dev/null @@ -1,35 +0,0 @@ -There are a couple of basic settings that you use to tell the buildbot -what project it is working on. This information is used by status -reporters to let users find out more about the codebase being -exercised by this particular Buildbot installation. - -@example -c['projectName'] = "Buildbot" -c['projectURL'] = "http://buildbot.sourceforge.net/" -c['buildbotURL'] = "http://localhost:8010/" -@end example - -@bcindex c['projectName'] -@code{projectName} is a short string will be used to describe the -project that this buildbot is working on. For example, it is used as -the title of the waterfall HTML page. - -@bcindex c['projectURL'] -@code{projectURL} is a string that gives a URL for the project as a -whole. HTML status displays will show @code{projectName} as a link to -@code{projectURL}, to provide a link from buildbot HTML pages to your -project's home page. - -@bcindex c['buildbotURL'] -The @code{buildbotURL} string should point to the location where the buildbot's -internal web server is visible. This typically uses the port number set when -you create the @code{Waterfall} object: the buildbot needs your help to figure -out a suitable externally-visible host name. - -When status notices are sent to users (either by email or over IRC), -@code{buildbotURL} will be used to create a URL to the specific build -or problem that they are being notified about. It will also be made -available to queriers (over IRC) who want to find out where to get -more information about this buildbot. - - diff --git a/docs/configuration.texinfo b/docs/configuration.texinfo index 31cef0568d9..55aeed37588 100644 --- a/docs/configuration.texinfo +++ b/docs/configuration.texinfo @@ -22,12 +22,9 @@ understand how to fill in each section properly. * Predefined Config File Symbols:: * Loading the Config File:: * Testing the Config File:: -* Project Definitions:: * Global Configuration:: * Change Sources:: * Schedulers:: -* Merging BuildRequests:: -* Prioritizing Builders:: * Setting the slaveport:: * Buildslave Specifiers:: * On-Demand ("Latent") Buildslaves:: @@ -179,10 +176,6 @@ NameError: name 'bogus' is not defined @end example -@node Project Definitions -@section Project Definitions -@include cfg-projdef.texinfo - @node Global Configuration @section Global Configuration @include cfg-global.texinfo @@ -195,67 +188,7 @@ NameError: name 'bogus' is not defined @section Schedulers @include cfg-schedulers.texinfo -@node Merging BuildRequests -@section Merging BuildRequests - -@bcindex c['mergeRequests'] - -By default, buildbot merges BuildRequests that have the compatible -SourceStamps. This behaviour can be customized with the -@code{c['mergeRequests']} configuration key. This key specifies a function -which is called with three arguments: a @code{Builder} and two -@code{BuildRequest} objects. It should return true if the requests can be -merged. For example: - -@example -def mergeRequests(builder, req1, req2): - """Don't merge buildrequest at all""" - return False -c['mergeRequests'] = mergeRequests -@end example - -In many cases, the details of the SourceStamps and BuildRequests are important. -In this example, only BuildRequests with the same "reason" are merged; thus -developers forcing builds for different reasons will see distinct builds. - -@example -def mergeRequests(builder, req1, req2): - if req1.source.canBeMergedWith(req2.source) and req1.reason == req2.reason: - return True - return False -c['mergeRequests'] = mergeRequests -@end example - -@node Prioritizing Builders -@section Prioritizing Builders - -@bcindex c['prioritizeBuilders'] - -By default, buildbot will attempt to start builds on builders in order from the -builder with the oldest pending request to the newest. This behaviour can be -customized with the @code{c['prioritizeBuilders']} configuration key. -This key specifies a function which is called with two arguments: a -@code{BuildMaster} and a list of @code{Builder} objects. It -should return a list of @code{Builder} objects in the desired order. -It may also remove items from the list if builds should not be started -on those builders. - -@example -def prioritizeBuilders(buildmaster, builders): - """Prioritize builders. 'finalRelease' builds have the highest - priority, so they should be built before running tests, or - creating builds.""" - builderPriorities = @{ - "finalRelease": 0, - "test": 1, - "build": 2, - @} - builders.sort(key=lambda b: builderPriorities.get(b.name, 0)) - return builders - -c['prioritizeBuilders'] = prioritizeBuilders -@end example - +@c working here -- @node Setting the slaveport @section Setting the slaveport