From e2f35482f17e1bf188c710b3d346af19e3537dcb Mon Sep 17 00:00:00 2001 From: Mikhail Sobolev Date: Sun, 3 Aug 2014 18:14:51 +0300 Subject: [PATCH] make formatting related changes to the affected documents * this will help the rebase should they have changed in the base branch * do some whitespace cleanup (trailing whitespaces, empty lines) * move change hooks to the end of customization.rst (relies on non-existent feature, hence it's not clear how to use it) * a bit of extra formatting (not part of this change, but anyway) --- master/docs/developer/index.rst | 7 +- master/docs/manual/cfg-builders.rst | 9 +- master/docs/manual/cfg-buildfactories.rst | 21 +- master/docs/manual/cfg-buildslaves.rst | 106 +++-- master/docs/manual/cfg-buildsteps.rst | 257 ++++++----- master/docs/manual/cfg-changesources.rst | 53 ++- master/docs/manual/cfg-interlocks.rst | 37 +- master/docs/manual/cfg-properties.rst | 34 +- master/docs/manual/cfg-schedulers.rst | 50 +- master/docs/manual/cfg-statustargets.rst | 527 +++++++++++----------- master/docs/manual/cfg-www.rst | 15 +- master/docs/manual/customization.rst | 52 ++- master/docs/tutorial/fiveminutes.rst | 2 +- 13 files changed, 616 insertions(+), 554 deletions(-) diff --git a/master/docs/developer/index.rst b/master/docs/developer/index.rst index 7e81bb371db..8bc6c0e1f21 100644 --- a/master/docs/developer/index.rst +++ b/master/docs/developer/index.rst @@ -3,10 +3,8 @@ Buildbot Development ==================== -This chapter is the official repository for the collected wisdom of the -Buildbot hackers. It is intended both for developers writing patches that will -be included in Buildbot itself, and for advanced users who wish to customize -Buildbot. +This chapter is the official repository for the collected wisdom of the Buildbot hackers. +It is intended both for developers writing patches that will be included in Buildbot itself, and for advanced users who wish to customize Buildbot. .. toctree:: :maxdepth: 2 @@ -14,4 +12,3 @@ Buildbot. general apis classes - diff --git a/master/docs/manual/cfg-builders.rst b/master/docs/manual/cfg-builders.rst index 99101522414..334a5526fb5 100644 --- a/master/docs/manual/cfg-builders.rst +++ b/master/docs/manual/cfg-builders.rst @@ -11,11 +11,10 @@ Builder Configuration :depth: 1 :local: -The :bb:cfg:`builders` configuration key is a list of objects giving -configuration for the Builders. For more information on the function of -Builders in Buildbot, see :ref:`the Concepts chapter `. The class -definition for the builder configuration is in :file:`buildbot.config`. In the -configuration file, its use looks like:: +The :bb:cfg:`builders` configuration key is a list of objects giving configuration for the Builders. +For more information on the function of Builders in Buildbot, see :ref:`the Concepts chapter `. +The class definition for the builder configuration is in :file:`buildbot.config`. +However there is a much simpler way to use it, so in the configuration file, its use looks like:: from buildbot.config import BuilderConfig c['builders'] = [ diff --git a/master/docs/manual/cfg-buildfactories.rst b/master/docs/manual/cfg-buildfactories.rst index b7611459335..5f5c6bf5a14 100644 --- a/master/docs/manual/cfg-buildfactories.rst +++ b/master/docs/manual/cfg-buildfactories.rst @@ -41,15 +41,15 @@ This factory would then be attached to one builder (or several, if desired):: c['builders'].append( BuilderConfig(name='quick', slavenames=['bot1', 'bot2'], factory=f)) -It is also possible to pass a list of steps into the -:class:`BuildFactory` when it is created. Using :meth:`addStep` is -usually simpler, but there are cases where it is more convenient to -create the list of steps ahead of time, perhaps using some Python -tricks to generate the steps. :: +It is also possible to pass a list of steps into the :class:`BuildFactory` when it is created. +Using :meth:`addStep` is usually simpler, but there are cases where it is more convenient to create the list of steps ahead of time, perhaps using some Python tricks to generate the steps. + +:: from buildbot.steps import source, shell from buildbot.process import factory + all_steps = [ source.CVS(cvsroot=CVSROOT, cvsmodule="project", mode="update"), shell.Compile(command=["make", "build"]), @@ -63,7 +63,9 @@ Finally, you can also add a sequence of steps all at once:: Attributes ~~~~~~~~~~ -The following attributes can be set on a build factory after it is created, e.g., :: +The following attributes can be set on a build factory after it is created, e.g., + +:: f = factory.BuildFactory() f.useProgress = False @@ -428,9 +430,6 @@ Arguments: run the test cases in random order, which sometimes catches subtle inter-test dependency bugs. Defaults to ``False``. -The step can also take any of the :class:`ShellCommand` arguments, e.g., -:attr:`haltOnFailure`. - -Unless one of ``tests`` or ``testChanges`` are set, the step will -generate an exception. +The step can also take any of the :class:`ShellCommand` arguments, e.g., :attr:`haltOnFailure`. +Unless one of ``tests`` or ``testChanges`` are set, the step will generate an exception. diff --git a/master/docs/manual/cfg-buildslaves.rst b/master/docs/manual/cfg-buildslaves.rst index a59d85cdca6..56491ae2bef 100644 --- a/master/docs/manual/cfg-buildslaves.rst +++ b/master/docs/manual/cfg-buildslaves.rst @@ -182,6 +182,11 @@ The following options are available for all latent buildslaves. Amazon Web Services Elastic Compute Cloud ("AWS EC2") +++++++++++++++++++++++++++++++++++++++++++++++++++++ +.. + XXX(sa2ajj): is it a good candidate to be put into a separate file? + + The point is: tell about availability, but the detailed guide in a separate doc? + `EC2 `_ is a web service that allows you to start virtual machines in an Amazon data center. Please see their website for details, including costs. Using the AWS EC2 latent buildslaves involves getting @@ -261,8 +266,10 @@ of this writing include ``m1.small``, ``m1.large``, ``m1.xlarge``, ``c1.medium`` and ``c1.xlarge``; see the EC2 documentation for descriptions of these machines). -Here is the simplest example of configuring an EC2 latent buildslave. It -specifies all necessary remaining values explicitly in the instantiation. :: +Here is the simplest example of configuring an EC2 latent buildslave. +It specifies all necessary remaining values explicitly in the instantiation. + +:: from buildbot.buildslave.ec2 import EC2LatentBuildSlave c['slaves'] = [EC2LatentBuildSlave('bot1', 'sekrit', 'm1.large', @@ -283,11 +290,12 @@ the account information can be specified in alternate ways. separate file, and potentially let more people read your main configuration file. -By default, you can make an :file:`.ec2` directory in the home folder of the user -running the buildbot master. In that directory, create a file called :file:`aws_id`. -The first line of that file should be your access key id; the second line -should be your secret access key id. Then you can instantiate the build slave -as follows. :: +By default, you can make an :file:`.ec2` directory in the home folder of the user running the buildbot master. +In that directory, create a file called :file:`aws_id`. +The first line of that file should be your access key id; the second line should be your secret access key id. +Then you can instantiate the build slave as follows. + +:: from buildbot.buildslave.ec2 import EC2LatentBuildSlave c['slaves'] = [EC2LatentBuildSlave('bot1', 'sekrit', 'm1.large', @@ -304,9 +312,9 @@ specify one or two AMI filters. In all cases, the AMI that sorts last by its location (the S3 bucket and manifest name) will be preferred. -One available filter is to specify the acceptable AMI owners, by AWS account -number (the 12 digit number, usually rendered in AWS with hyphens like -"1234-5678-9012", should be entered as in integer). :: +One available filter is to specify the acceptable AMI owners, by AWS account number (the 12 digit number, usually rendered in AWS with hyphens like "1234-5678-9012", should be entered as in integer). + +:: from buildbot.buildslave.ec2 import EC2LatentBuildSlave bot1 = EC2LatentBuildSlave('bot1', 'sekrit', 'm1.large', @@ -316,8 +324,9 @@ number (the 12 digit number, usually rendered in AWS with hyphens like secret_identifier='privatekey' ) -The other available filter is to provide a regular expression string that -will be matched against each AMI's location (the S3 bucket and manifest name). :: +The other available filter is to provide a regular expression string that will be matched against each AMI's location (the S3 bucket and manifest name). + +:: from buildbot.buildslave.ec2 import EC2LatentBuildSlave bot1 = EC2LatentBuildSlave( @@ -325,8 +334,10 @@ will be matched against each AMI's location (the S3 bucket and manifest name). : valid_ami_location_regex=r'buildbot\-.*/image.manifest.xml', identifier='publickey', secret_identifier='privatekey') -The regular expression can specify a group, which will be preferred for the -sorting. Only the first group is used; subsequent groups are ignored. :: +The regular expression can specify a group, which will be preferred for the sorting. +Only the first group is used; subsequent groups are ignored. + +:: from buildbot.buildslave.ec2 import EC2LatentBuildSlave bot1 = EC2LatentBuildSlave( @@ -334,8 +345,10 @@ sorting. Only the first group is used; subsequent groups are ignored. :: valid_ami_location_regex=r'buildbot\-.*\-(.*)/image.manifest.xml', identifier='publickey', secret_identifier='privatekey') -If the group can be cast to an integer, it will be. This allows 10 to sort -after 1, for instance. :: +If the group can be cast to an integer, it will be. +This allows 10 to sort after 1, for instance. + +:: from buildbot.buildslave.ec2 import EC2LatentBuildSlave bot1 = EC2LatentBuildSlave( @@ -343,11 +356,11 @@ after 1, for instance. :: valid_ami_location_regex=r'buildbot\-.*\-(\d+)/image.manifest.xml', identifier='publickey', secret_identifier='privatekey') -In addition to using the password as a handshake between the master and the -slave, you may want to use a firewall to assert that only machines from a -specific IP can connect as slaves. This is possible with AWS EC2 by using -the Elastic IP feature. To configure, generate a Elastic IP in AWS, and then -specify it in your configuration using the ``elastic_ip`` argument. :: +In addition to using the password as a handshake between the master and the slave, you may want to use a firewall to assert that only machines from a specific IP can connect as slaves. +This is possible with AWS EC2 by using the Elastic IP feature. +To configure, generate a Elastic IP in AWS, and then specify it in your configuration using the ``elastic_ip`` argument. + +:: from buildbot.buildslave.ec2 import EC2LatentBuildSlave c['slaves'] = [EC2LatentBuildSlave('bot1', 'sekrit', 'm1.large', @@ -357,9 +370,11 @@ specify it in your configuration using the ``elastic_ip`` argument. :: elastic_ip='208.77.188.166' )] -One other way to configure a slave is by settings AWS tags. They can for example be used to -have a more restrictive security `IAM `_ policy. To get Buildbot to tag the latent slave -specify the tag keys and values in your configuration using the ``tags`` argument. :: +One other way to configure a slave is by settings AWS tags. +They can for example be used to have a more restrictive security `IAM `_ policy. +To get Buildbot to tag the latent slave specify the tag keys and values in your configuration using the ``tags`` argument. + +:: from buildbot.buildslave.ec2 import EC2LatentBuildSlave c['slaves'] = [EC2LatentBuildSlave('bot1', 'sekrit', 'm1.large', @@ -384,10 +399,10 @@ these AWS EC2 values. They both default to ``latent_buildbot_slave``. Spot instances ############## -If you would prefer to use spot instances for running your builds, you can accomplish that -by passing in a True value to the ``spot_instance`` parameter to the EC2LatentBuildSlave -constructor. Additionally, you may want to specify ``max_spot_price`` and ``price_multiplier`` -in order to limit your builds' budget consumption. :: +If you would prefer to use spot instances for running your builds, you can accomplish that by passing in a True value to the ``spot_instance`` parameter to the EC2LatentBuildSlave constructor. +Additionally, you may want to specify ``max_spot_price`` and ``price_multiplier`` in order to limit your builds' budget consumption. + +:: from buildbot.buildslave.ec2 import EC2LatentBuildSlave c['slaves'] = [EC2LatentBuildSlave('bot1', 'sekrit', 'm1.large', @@ -409,12 +424,11 @@ parameter, then a spot request will be sent to Amazon with the above details. When a spot request fails ######################### -In some cases Amazon may reject a spot request because the spot price, determined by taking -the 24-hour average of that availability zone's spot prices for the given product description, -is lower than the current price. The optional parameters ``retry`` and ``retry_price_adjustment`` -allow for resubmitting the spot request with an adjusted price. If the spot request continues to -fail, and the number of attempts exceeds the value of the ``retry`` parameter, an error message -will be logged. :: +In some cases Amazon may reject a spot request because the spot price, determined by taking the 24-hour average of that availability zone's spot prices for the given product description, is lower than the current price. +The optional parameters ``retry`` and ``retry_price_adjustment`` allow for resubmitting the spot request with an adjusted price. +If the spot request continues to fail, and the number of attempts exceeds the value of the ``retry`` parameter, an error message will be logged. + +:: from buildbot.buildslave.ec2 import EC2LatentBuildSlave c['slaves'] = [EC2LatentBuildSlave('bot1', 'sekrit', 'm1.large', @@ -441,6 +455,9 @@ as normal and run any pending builds. Libvirt +++++++ +.. + XXX(sa2ajj): Ditto + `libvirt `_ is a virtualization API for interacting with the virtualization capabilities of recent versions of Linux and other OSes. It is LGPL and comes with a stable C API, and Python bindings. @@ -508,7 +525,9 @@ Configuring your Master If you want to add a simple on demand VM to your setup, you only need the following. We set the username to ``minion1``, the password to ``sekrit``. The base image is called ``base_image`` and a copy of it will be made for the duration of the VM's life. That copy will be thrown -away every time a build is complete. :: +away every time a build is complete. + +:: from buildbot.buildslave.libvirt import LibVirtSlave, Connection c['slaves'] = [LibVirtSlave('minion1', 'sekrit', Connection("qemu:///session"), @@ -542,6 +561,10 @@ won't be able to find a VM to start. OpenStack +++++++++ + +.. + XXX(sa2ajj): Ditto + `OpenStack `_ is a series of interconnected components that facilitates managing compute, storage, and network resources in a data center. It is available under the Apache License and has a REST interface @@ -599,7 +622,9 @@ passed as options to an OpenStack client. A dictionary of string key-value pairs to pass to the instance. These will be available under the ``metadata`` key from the metadata service. -Here is the simplest example of configuring an OpenStack latent buildslave. :: +Here is the simplest example of configuring an OpenStack latent buildslave. + +:: from buildbot.buildslave.openstack import OpenStackLatentBuildSlave c['slaves'] = [OpenStackLatentBuildSlave('bot2', 'sekrit', @@ -608,10 +633,11 @@ Here is the simplest example of configuring an OpenStack latent buildslave. :: os_tenant_name='tenant', os_auth_url='http://127.0.0.1:35357/v2.0')] -The ``image`` argument also supports being given a callable. The callable will -be passed the list of available images and must return the image to use. The -invocation happens in a separate thread to prevent blocking the build master -when interacting with OpenStack. :: +The ``image`` argument also supports being given a callable. +The callable will be passed the list of available images and must return the image to use. +The invocation happens in a separate thread to prevent blocking the build master when interacting with OpenStack. + +:: from buildbot.buildslave.openstack import OpenStackLatentBuildSlave diff --git a/master/docs/manual/cfg-buildsteps.rst b/master/docs/manual/cfg-buildsteps.rst index c1f932a0059..4902d3900c8 100644 --- a/master/docs/manual/cfg-buildsteps.rst +++ b/master/docs/manual/cfg-buildsteps.rst @@ -159,7 +159,9 @@ Arguments common to all :class:`BuildStep` subclasses: A step can be optionally hidden from the waterfall and build details web pages. To do this, set the step's ``hideStepIf`` to a boolean value, or to a function that takes two parameters -- the results and the :class:`BuildStep` -- and returns a boolean value. Steps are always shown while they execute, however after the step as finished, this parameter is evaluated (if a function) and if the value is True, the step is hidden. - For example, in order to hide the step if the step has been skipped, :: + For example, in order to hide the step if the step has been skipped, + + :: factory.addStep(Foo(..., hideStepIf=lambda results, s: results==SKIPPED)) @@ -185,15 +187,15 @@ Source Checkout .. py:module:: buildbot.steps.source -.. caution:: Support for the old slave-side source checkout steps was removed - in Buildbot-0.9.0. +.. caution:: + + Support for the old slave-side source checkout steps was removed in Buildbot-0.9.0. The old source steps were imported like this:: from buildbot.steps.source import Git - while new source steps are in separate Python modules for each - version-control system:: + while new source steps are in separate Python modules for each version-control system:: from buildbot.steps.source.git import Git @@ -209,8 +211,9 @@ parameters are mostly to specify where exactly the sources are coming from. ``method`` These two parameters specify the means by which the source is checked out. - ``mode`` specifies the type of checkout and ``method`` tells about the - way to implement it. :: + ``mode`` specifies the type of checkout and ``method`` tells about the way to implement it. + + :: factory = BuildFactory() from buildbot.steps.source.mercurial import Mercurial @@ -301,7 +304,9 @@ The :bb:step:`Mercurial` build step performs a `Mercurial `_ -repository and checks out the specified branch or revision. Note that -the buildbot supports Git version 1.2.0 and later: earlier versions -(such as the one shipped in Ubuntu 'Dapper') do not support the -:command:`git init` command that the buildbot uses. :: +The ``Git`` build step clones or updates a `Git `_ repository and checks out the specified branch or revision. +Note that the buildbot supports Git version 1.2.0 and later: earlier versions (such as the one shipped in Ubuntu 'Dapper') do not support the :command:`git init` command that the buildbot uses. + +:: from buildbot.steps.source.git import Git factory.addStep(Git(repourl='git://path/to/repo', mode='full', @@ -614,8 +618,9 @@ CVS .. py:class:: buildbot.steps.source.cvs.CVS -The :bb:step:`CVS` build step performs a `CVS `_ -checkout or update. :: +The :bb:step:`CVS` build step performs a `CVS `_ checkout or update. + +:: from buildbot.steps.source.cvs import CVS factory.addStep(CVS(mode='incremental', @@ -679,11 +684,12 @@ Bzr .. py:class:: buildbot.steps.source.bzr.Bzr -bzr is a descendant of Arch/Baz, and is frequently referred to -as simply `Bazaar`. The repository-vs-workspace model is similar to -Darcs, but it uses a strictly linear sequence of revisions (one -history per branch) like Arch. Branches are put in subdirectories. -This makes it look very much like Mercurial. :: +bzr is a descendant of Arch/Baz, and is frequently referred to as simply `Bazaar`. +The repository-vs-workspace model is similar to Darcs, but it uses a strictly linear sequence of revisions (one history per branch) like Arch. +Branches are put in subdirectories. +This makes it look very much like Mercurial. + +:: from buildbot.steps.source.bzr import Bzr factory.addStep(Bzr(mode='incremental', @@ -738,8 +744,9 @@ P4 .. py:class:: buildbot.steps.source.p4.P4 -The :bb:step:`P4` build step creates a `Perforce `_ -client specification and performs an update. :: +The :bb:step:`P4` build step creates a `Perforce `_ client specification and performs an update. + +:: from buildbot.steps.source.p4 import P4 factory.addStep(P4(p4port=p4port, @@ -987,8 +994,9 @@ Darcs .. py:class:: buildbot.steps.source.darcs.Darcs -The :bb:step:`Darcs` build step performs a `Darcs `_ -checkout or update. :: +The :bb:step:`Darcs` build step performs a `Darcs `_ checkout or update. + +:: from buildbot.steps.source.darcs import Darcs factory.addStep(Darcs(repourl='http://path/to/repo', @@ -1039,8 +1047,9 @@ Monotone .. py:class:: buildbot.steps.source.mtn.Monotone -The :bb:step:`Monotone ` build step performs a `Monotone `_ -checkout or update. :: +The :bb:step:`Monotone ` build step performs a `Monotone `_ checkout or update. + +:: from buildbot.steps.source.mtn import Monotone factory.addStep(Darcs(repourl='http://path/to/repo', @@ -1087,7 +1096,7 @@ Monotone step takes the following arguments: ``clean`` This remove all other files except those tracked and ignored by Monotone. It will remove - all the files that appear in :command:`mtn ls unknown`. Then it will pull from + all the files that appear in :command:`mtn ls unknown`. Then it will pull from remote and update the working directory. ``fresh`` @@ -1175,7 +1184,7 @@ The :bb:step:`ShellCommand` arguments are: ``env`` a dictionary of environment strings which will be added to the child command's environment. For example, to run tests with a different i18n - language setting, you might use :: + language setting, you might use:: from buildbot.steps.shell import ShellCommand f.addStep(ShellCommand(command=["make", "test"], @@ -1202,7 +1211,9 @@ The :bb:step:`ShellCommand` arguments are: :file:`/home/buildbot/bin` to the :envvar:`PATH` environment variable, you can do it by putting the value ``${PATH}`` at the end of the value like in the example below. Variables that don't exist on the slave will be - replaced by ``""``. :: + replaced by ``""``. + + :: from buildbot.steps.shell import ShellCommand f.addStep(ShellCommand( @@ -1264,21 +1275,21 @@ The :bb:step:`ShellCommand` arguments are: of seconds) as the build runs, and any new text will be sent over to the buildmaster. - If you provide a dictionary of options instead of a string, you must specify - the ``filename`` key. You can optionally provide a ``follow`` key which - is a boolean controlling whether a logfile is followed or concatenated in its - entirety. Following is appropriate for logfiles to which the build step will - append, where the pre-existing contents are not interesting. The default value - for ``follow`` is ``False``, which gives the same behavior as just - providing a string filename. :: + If you provide a dictionary of options instead of a string, you must specify the ``filename`` key. + You can optionally provide a ``follow`` key which is a boolean controlling whether a logfile is followed or concatenated in its entirety. + Following is appropriate for logfiles to which the build step will append, where the pre-existing contents are not interesting. + The default value for ``follow`` is ``False``, which gives the same behavior as just providing a string filename. + + :: from buildbot.steps.shell import ShellCommand f.addStep(ShellCommand( command=["make", "test"], logfiles={"triallog": "_trial_temp/test.log"})) - The above example will add a log named 'triallog' on the master, - based on :file:`_trial_temp/test.log` on the slave. :: + The above example will add a log named 'triallog' on the master, based on :file:`_trial_temp/test.log` on the slave. + + :: from buildbot.steps.shell import ShellCommand f.addStep(ShellCommand( @@ -1355,8 +1366,9 @@ The single :bb:step:`ShellSequence` argument aside from the common parameters is ``commands`` -A list of :class:`~buildbot.steps.shellsequence.ShellArg` objects or a renderable the returns -a list of :class:`~buildbot.steps.shellsequence.ShellArg` objects. :: +A list of :class:`~buildbot.steps.shellsequence.ShellArg` objects or a renderable the returns a list of :class:`~buildbot.steps.shellsequence.ShellArg` objects. + +:: from buildbot.steps.shellsequence import ShellArg from buildbot.steps.shellsequence import ShellSequence @@ -1416,7 +1428,9 @@ This is intended to handle the :command:`./configure` step from autoconf-style projects, or the ``perl Makefile.PL`` step from perl :file:`MakeMaker.pm`-style modules. The default command is :command:`./configure` but you can change this by providing a ``command=`` parameter. The arguments are -identical to :bb:step:`ShellCommand`. :: +identical to :bb:step:`ShellCommand`. + +:: from buildbot.steps.shell import Configure f.addStep(Configure()) @@ -1855,8 +1869,9 @@ SubunitShellCommand .. py:class:: buildbot.steps.subunit.SubunitShellCommand -This buildstep is similar to :bb:step:`ShellCommand`, except that it runs the log content -through a subunit filter to extract test and failure counts. :: +This buildstep is similar to :bb:step:`ShellCommand`, except that it runs the log content through a subunit filter to extract test and failure counts. + +:: from buildbot.steps.subunit import SubunitShellCommand f.addStep(SubunitShellCommand(command="make test")) @@ -1879,8 +1894,10 @@ Here are some buildsteps for manipulating the slave's filesystem. FileExists ++++++++++ -This step will assert that a given file exists, failing if it does not. The -filename can be specified with a property. :: +This step will assert that a given file exists, failing if it does not. +The filename can be specified with a property. + +:: from buildbot.steps.slave import FileExists f.addStep(FileExists(file='test_data')) @@ -1892,7 +1909,9 @@ This step requires slave version 0.8.4 or later. CopyDirectory +++++++++++++++ -This command copies a directory on the slave. :: +This command copies a directory on the slave. + +:: from buildbot.steps.slave import CopyDirectory f.addStep(CopyDirectory(src="build/data", dest="tmp/data")) @@ -1915,7 +1934,9 @@ The CopyDirectory step takes the following arguments: RemoveDirectory +++++++++++++++ -This command recursively deletes a directory on the slave. :: +This command recursively deletes a directory on the slave. + +:: from buildbot.steps.slave import RemoveDirectory f.addStep(RemoveDirectory(dir="build/build")) @@ -1927,7 +1948,9 @@ This step requires slave version 0.8.4 or later. MakeDirectory +++++++++++++++ -This command creates a directory on the slave. :: +This command creates a directory on the slave. + +:: from buildbot.steps.slave import MakeDirectory f.addStep(MakeDirectory(dir="build/build")) @@ -1968,13 +1991,12 @@ with an `epydocs` target. You might wish to use something like to add :option:`--pdf` to generate a PDF file instead of a large tree of HTML files. -The API docs are generated in-place in the build tree (under the -workdir, in the subdirectory controlled by the :option:`-o` argument). To -make them useful, you will probably have to copy them to somewhere -they can be read. A command like ``rsync -ad apiref/ -dev.example.com:~public_html/current-apiref/`` might be useful. You -might instead want to bundle them into a tarball and publish it in the -same place where the generated install tarball is placed. :: +The API docs are generated in-place in the build tree (under the workdir, in the subdirectory controlled by the :option:`-o` argument). +To make them useful, you will probably have to copy them to somewhere they can be read. +A command like ``rsync -ad apiref/ dev.example.com:~public_html/current-apiref/`` might be useful. +You might instead want to bundle them into a tarball and publish it in the same place where the generated install tarball is placed. + +:: from buildbot.steps.python import BuildEPYDoc f.addStep(BuildEPYDoc(command=["epydoc", "-o", "apiref", "source/mypkg"])) @@ -1998,10 +2020,11 @@ provide more detailed results but take longer to run. The :bb:step:`PyFlakes` step will run pyflakes and count the various kinds of errors and warnings it detects. -You must supply the command line to be used. The default is -``make pyflakes``, which assumes you have a top-level :file:`Makefile` -with a ``pyflakes`` target. You might want to use something like -``pyflakes .`` or ``pyflakes src``. :: +You must supply the command line to be used. +The default is ``make pyflakes``, which assumes you have a top-level :file:`Makefile` with a ``pyflakes`` target. +You might want to use something like ``pyflakes .`` or ``pyflakes src``. + +:: from buildbot.steps.python import PyFlakes f.addStep(PyFlakes(command=["pyflakes", "src"])) @@ -2019,10 +2042,9 @@ Sphinx Generator. It uses `RestructuredText `_ as input format. -The :bb:step:`Sphinx` step will run -:program:`sphinx-build` or any other program specified in its -``sphinx`` argument and count the various warnings and error it -detects. :: +The :bb:step:`Sphinx` step will run :program:`sphinx-build` or any other program specified in its ``sphinx`` argument and count the various warnings and error it detects. + +:: from buildbot.steps.python import Sphinx f.addStep(Sphinx(sphinx_builddir="_build")) @@ -2065,7 +2087,10 @@ PyLint Similarly, the :bb:step:`PyLint` step will run :command:`pylint` and analyze the results. -You must supply the command line to be used. There is no default. :: +You must supply the command line to be used. +There is no default. + +:: from buildbot.steps.python import PyLint f.addStep(PyLint(command=["pylint", "src"])) @@ -2127,10 +2152,10 @@ To influence the version of Python being used for the tests, or to add flags to the command, set the ``python`` parameter. This can be a string (like ``python2.2``) or a list (like ``['python2.3', '-Wall']``). -Trial creates and switches into a directory named :file:`_trial_temp/` before -running the tests, and sends the twisted log (which includes all exceptions) to -a file named :file:`test.log`. This file will be pulled up to the master where -it can be seen as part of the status output. :: +Trial creates and switches into a directory named :file:`_trial_temp/` before running the tests, and sends the twisted log (which includes all exceptions) to a file named :file:`test.log`. +This file will be pulled up to the master where it can be seen as part of the status output. + +:: from buildbot.steps.python_twisted import Trial f.addStep(Trial(tests='petmail.test')) @@ -2155,11 +2180,11 @@ RemovePYCs .. py:class:: buildbot.steps.python_twisted.RemovePYCs -This is a simple built-in step that will remove ``.pyc`` files from the -workdir. This is useful in builds that update their source (and thus do not -automatically delete ``.pyc`` files) but where some part of the build -process is dynamically searching for Python modules. Notably, trial has a bad -habit of finding old test modules. :: +This is a simple built-in step that will remove ``.pyc`` files from the workdir. +This is useful in builds that update their source (and thus do not automatically delete ``.pyc`` files) but where some part of the build process is dynamically searching for Python modules. +Notably, trial has a bad habit of finding old test modules. + +:: from buildbot.steps.python_twisted import RemovePYCs f.addStep(RemovePYCs()) @@ -2184,14 +2209,12 @@ There are a pair of steps named :bb:step:`FileUpload` and moves a file *up to* the master, while :bb:step:`FileDownload` moves a file *down from* the master. -As an example, let's assume that there is a step which produces an -HTML file within the source tree that contains some sort of generated -project documentation. We want to move this file to the buildmaster, -into a :file:`~/public_html` directory, so it can be visible to -developers. This file will wind up in the slave-side working directory -under the name :file:`docs/reference.html`. We want to put it into the -master-side :file:`~/public_html/ref.html`, and add a link to the HTML -status to the uploaded file. :: +As an example, let's assume that there is a step which produces an HTML file within the source tree that contains some sort of generated project documentation. +We want to move this file to the buildmaster, into a :file:`~/public_html` directory, so it can be visible to developers. +This file will wind up in the slave-side working directory under the name :file:`docs/reference.html`. +We want to put it into the master-side :file:`~/public_html/ref.html`, and add a link to the HTML status to the uploaded file. + +:: from buildbot.steps.shell import ShellCommand from buildbot.steps.transfer import FileUpload @@ -2375,16 +2398,18 @@ Transfering Strings .. py:class:: buildbot.steps.transfer.JSONStringDownload .. py:class:: buildbot.steps.transfer.JSONPropertiesDownload -Sometimes it is useful to transfer a calculated value from the master to the -slave. Instead of having to create a temporary file and then use FileDownload, -you can use one of the string download steps. :: +Sometimes it is useful to transfer a calculated value from the master to the slave. +Instead of having to create a temporary file and then use FileDownload, you can use one of the string download steps. + +:: from buildbot.steps.transfer import StringDownload f.addStep(StringDownload(Interpolate("%(src::branch)s-%(prop:got_revision)s\n"), slavedest="buildid.txt")) -:bb:step:`StringDownload` works just like :bb:step:`FileDownload` except it takes a single argument, -``s``, representing the string to download instead of a ``mastersrc`` argument. :: +:bb:step:`StringDownload` works just like :bb:step:`FileDownload` except it takes a single argument, ``s``, representing the string to download instead of a ``mastersrc`` argument. + +:: from buildbot.steps.transfer import JSONStringDownload buildinfo = { branch: Property('branch'), got_revision: Property('got_revision') } @@ -2421,7 +2446,9 @@ the master, instead of the slave. To be clear, the enclosing :class:`Build` object must still have a slave object, just as for any other step -- only, in this step, the slave does not do anything. -In this example, the step renames a tarball based on the day of the week. :: +In this example, the step renames a tarball based on the day of the week. + +:: from buildbot.steps.transfer import FileUpload from buildbot.steps.master import MasterShellCommand @@ -2440,7 +2467,9 @@ Environment variables constructed using the ``env`` argument support expansion so that if you just want to prepend :file:`/home/buildbot/bin` to the :envvar:`PATH` environment variable, you can do it by putting the value ``${PATH}`` at the end of the value like in the example below. -Variables that don't exist on the master will be replaced by ``""``. :: +Variables that don't exist on the master will be replaced by ``""``. + +:: from buildbot.steps.master import MasterShellCommand f.addStep(MasterShellCommand( @@ -2512,21 +2541,21 @@ add ``strip=False``. The ``property`` argument can be specified as a :ref:`Interpolate` object, allowing the property name to be built from other property values. -The more advanced usage allows you to specify a function to extract -properties from the command output. Here you can use regular -expressions, string interpolation, or whatever you would like. In this -form, :func:`extract_fn` should be passed, and not :class:`Property`. -The :func:`extract_fn` function is called with three arguments: the exit status of the -command, its standard output as a string, and its standard error as -a string. It should return a dictionary containing all new properties. :: +The more advanced usage allows you to specify a function to extract properties from the command output. +Here you can use regular expressions, string interpolation, or whatever you would like. +In this form, :func:`extract_fn` should be passed, and not :class:`Property`. +The :func:`extract_fn` function is called with three arguments: the exit status of the command, its standard output as a string, and its standard error as a string. +It should return a dictionary containing all new properties. + +:: def glob2list(rc, stdout, stderr): jpgs = [ l.strip() for l in stdout.split('\n') ] return { 'jpgs' : jpgs } f.addStep(SetPropertyFromCommand(command="ls -1 *.jpg", extract_fn=glob2list)) -Note that any ordering relationship of the contents of stdout and -stderr is lost. For example, given :: +Note that any ordering relationship of the contents of stdout and stderr is lost. +For example, given:: f.addStep(SetPropertyFromCommand( command="echo output1; echo error >&2; echo output2", @@ -2550,12 +2579,11 @@ the :bb:step:`SetPropertiesFromEnv` step. Pass a variable or list of variables in the ``variables`` parameter, then simply use the values as properties in a later step. -Note that on Windows, environment variables are case-insensitive, but Buildbot -property names are case sensitive. The property will have exactly the variable -name you specify, even if the underlying environment variable is capitalized -differently. If, for example, you use ``variables=['Tmp']``, the result -will be a property named ``Tmp``, even though the environment variable is -displayed as :envvar:`TMP` in the Windows GUI. :: +Note that on Windows, environment variables are case-insensitive, but Buildbot property names are case sensitive. +The property will have exactly the variable name you specify, even if the underlying environment variable is capitalized differently. +If, for example, you use ``variables=['Tmp']``, the result will be a property named ``Tmp``, even though the environment variable is displayed as :envvar:`TMP` in the Windows GUI. + +:: from buildbot.steps.slave import SetPropertiesFromEnv from buildbot.steps.shell import Compile @@ -2623,7 +2651,7 @@ using properties. The ``set_properties`` parameter allows control of the properties that are passed to the triggered scheduler. The parameter takes a dictionary mapping property names to values. You may use :ref:`Interpolate` here to dynamically construct new property values. -For the simple case of copying a property, this might look like :: +For the simple case of copying a property, this might look like:: set_properties={"my_prop1" : Property("my_prop1")} @@ -2760,11 +2788,12 @@ Debian Build Steps DebPbuilder +++++++++++ -The :bb:step:`DebPbuilder` step builds Debian packages within a chroot built -by pbuilder. It populates the changeroot with a basic system and the packages -listed as build requirement. The type of chroot to build is specified with the -``distribution``, ``distribution`` and ``mirror`` parameter. To use pbuilder -your buildbot must have the right to run pbuilder as root through sudo. :: +The :bb:step:`DebPbuilder` step builds Debian packages within a chroot built by pbuilder. +It populates the changeroot with a basic system and the packages listed as build requirement. +The type of chroot to build is specified with the ``distribution``, ``distribution`` and ``mirror`` parameter. +To use pbuilder your buildbot must have the right to run pbuilder as root through sudo. + +:: from buildbot.steps.package.deb.pbuilder import DebPbuilder f.addStep(DebPbuilder()) @@ -2832,8 +2861,10 @@ The step looks at the list of changes in the build to determine which files to check - it does not check all files. It specifically excludes any ``.xhtml`` files in the top-level ``sandbox/`` directory. -The step takes a single, optional, parameter: ``python``. This specifies the -Python executable to use to run Lore. :: +The step takes a single, optional, parameter: ``python``. +This specifies the Python executable to use to run Lore. + +:: from buildbot.steps.python_twisted import HLint f.addStep(HLint()) @@ -2845,7 +2876,9 @@ MaxQ MaxQ (http://maxq.tigris.org/) is a web testing tool that allows you to record HTTP sessions and play them back. The :bb:step:`MaxQ` step runs this -framework. :: +framework. + +:: from buildbot.steps.maxq import MaxQ f.addStep(MaxQ(testdir='tests/')) diff --git a/master/docs/manual/cfg-changesources.rst b/master/docs/manual/cfg-changesources.rst index 8e651051393..9dd0f12b8c0 100644 --- a/master/docs/manual/cfg-changesources.rst +++ b/master/docs/manual/cfg-changesources.rst @@ -122,11 +122,11 @@ See :bb:index:`chsrc` for a full list of change sources. Configuring Change Sources ~~~~~~~~~~~~~~~~~~~~~~~~~~ -The :bb:cfg:`change_source` configuration key holds all active -change sources for the configuration. +The :bb:cfg:`change_source` configuration key holds all active change sources for the configuration. -Most configurations have a single :class:`ChangeSource`, watching only a single -tree, e.g., :: +Most configurations have a single :class:`ChangeSource`, watching only a single tree, e.g., + +:: c['change_source'] = PBChangeSource() @@ -179,8 +179,7 @@ change-emailing tools, and each has a slightly different format, so each has a different parsing function. There is a separate ChangeSource variant for each parsing function. -Once you've chosen a maildir location and a parsing function, create -the change source and put it in ``change_source`` :: +Once you've chosen a maildir location and a parsing function, create the change source and put it in ``change_source``:: from buildbot.changes.mail import CVSMaildirSource c['change_source'] = CVSMaildirSource("~/maildir-buildbot", @@ -385,8 +384,9 @@ SVNCommitEmailMaildirSource :file:`commit-email.pl` script, which is included in the Subversion distribution. -It does not currently handle branches: all of the Change objects that -it creates will be associated with the default (i.e. trunk) branch. :: +It does not currently handle branches: all of the Change objects that it creates will be associated with the default (i.e. trunk) branch. + +:: from buildbot.changes.mail import SVNCommitEmailMaildirSource c['change_source'] = SVNCommitEmailMaildirSource("~/maildir-buildbot") @@ -419,10 +419,11 @@ The ``prefix`` option is not supported (it is silently ignored). Use the branches (and just do not subscribe the buildbot to branches that are not of interest). -The revision number is obtained from the email text. The bzr revision id is -not available in the mails sent by Launchpad. However, it is possible to set -the bzr `append_revisions_only` option for public shared repositories to -avoid new pushes of merges changing the meaning of old revision numbers. :: +The revision number is obtained from the email text. +The bzr revision id is not available in the mails sent by Launchpad. +However, it is possible to set the bzr `append_revisions_only` option for public shared repositories to avoid new pushes of merges changing the meaning of old revision numbers. + +:: from buildbot.changes.mail import BzrLaunchpadEmailMaildirSource bm = { 'lp:~maria-captains/maria/5.1' : '5.1', 'lp:~maria-captains/maria/6.0' : '6.0' } @@ -631,10 +632,10 @@ depot for changes. It accepts the following arguments: Example +++++++ -This configuration uses the :envvar:`P4PORT`, :envvar:`P4USER`, and :envvar:`P4PASSWD` -specified in the buildmaster's environment. It watches a project in which the -branch name is simply the next path component, and the file is all path -components after. :: +This configuration uses the :envvar:`P4PORT`, :envvar:`P4USER`, and :envvar:`P4PASSWD` specified in the buildmaster's environment. +It watches a project in which the branch name is simply the next path component, and the file is all path components after. + +:: from buildbot.changes import p4poller s = p4poller.P4Source(p4base='//depot/project/', @@ -806,10 +807,7 @@ is available explicitly as ``split_file_alwaystrunk``:: svnurl="svn://svn.twistedmatrix.com/svn/Twisted/trunk", split_file=split_file_alwaystrunk) - -For repositories with the ``/trunk`` and -``/branches/{BRANCH}`` layout, ``split_file_branches`` will do the -job:: +For repositories with the ``/trunk`` and :samp:`/branches/{BRANCH}` layout, ``svn_split_file_branches`` will do the job:: from buildbot.changes.svnpoller import SVNPoller from buildbot.changes.svnpoller import split_file_branches @@ -820,9 +818,7 @@ job:: When using this splitter the poller will set the ``project`` attribute of any changes to the ``project`` attribute of the poller. -For repositories with the ``{PROJECT}/trunk`` and -``{PROJECT}/branches/{BRANCH}`` layout, ``split_file_projects_branches`` will do -the job:: +For repositories with the :samp:`{PROJECT}/trunk` and :samp:`{PROJECT}/branches/{BRANCH}` layout, ``svn_split_file_projects_branches`` will do the job:: from buildbot.changes.svnpoller import SVNPoller from buildbot.changes.svnpoller import split_file_projects_branches @@ -849,9 +845,11 @@ configuration can import it. Even putting it in the same directory as the :file: should work. Install the poller in the buildbot configuration as with any other change source. Minimally, provide a URL that you want to poll (``bzr://``, ``bzr+ssh://``, or ``lp:``), making sure the buildbot user has necessary -privileges. :: +privileges. + +:: - # bzr_buildbot.py in the same directory as master.cfg + # put bzr_buildbot.py file to the same directory as master.cfg from bzr_buildbot import BzrPoller c['change_source'] = BzrPoller( url='bzr://hostname/my_project', @@ -1347,5 +1345,6 @@ configuration would look like this:: feedurl="http://code.google.com/feeds/p/ostinato/hgchanges/basic", pollinterval=10800) -(note that you will need to download ``googlecode_atom.py`` from the Buildbot -source and install it somewhere on your PYTHONPATH first) +.. note:: + + You will need to download ``googlecode_atom.py`` from the Buildbot source and install it somewhere on your :envvar:`PYTHONPATH` first. diff --git a/master/docs/manual/cfg-interlocks.rst b/master/docs/manual/cfg-interlocks.rst index 87db0210fbd..a6ef6b70962 100644 --- a/master/docs/manual/cfg-interlocks.rst +++ b/master/docs/manual/cfg-interlocks.rst @@ -61,8 +61,10 @@ slave, like above. Examples ~~~~~~~~ -Time for a few examples. Below a master lock is defined to protect a data base, -and a slave lock is created to limit the number of builds at each slave. :: +Time for a few examples. +Below a master lock is defined to protect a data base, and a slave lock is created to limit the number of builds at each slave. + +:: from buildbot import locks @@ -71,15 +73,13 @@ and a slave lock is created to limit the number of builds at each slave. :: maxCount = 1, maxCountForSlave = { 'fast': 3, 'new': 2 }) -After importing locks from buildbot, :data:`db_lock` is defined to be a master -lock. The ``database`` string is used for uniquely identifying the lock. -At the next line, a slave lock called :data:`build_lock` is created. It is -identified by the ``slave_builds`` string. Since the requirements of the -lock are a bit more complicated, two optional arguments are also specified. The -``maxCount`` parameter sets the default limit for builds in counting mode to -``1``. For the slave called ``'fast'`` however, we want to have at most -three builds, and for the slave called ``'new'`` the upper limit is two -builds running at the same time. +:data:`db_lock` is defined to be a master lock. +The ``database`` string is used for uniquely identifying the lock. +At the next line, a slave lock called :data:`build_lock` is created. +It is identified by the ``slave_builds`` string. +Since the requirements of the lock are a bit more complicated, two optional arguments are also specified. +The ``maxCount`` parameter sets the default limit for builds in counting mode to ``1``. +For the slave called ``'fast'`` however, we want to have at most three builds, and for the slave called ``'new'`` the upper limit is two builds running at the same time. The next step is accessing the locks in builds. Buildbot allows a lock to be used during an entire build (from beginning to end), or only during a single build @@ -97,7 +97,9 @@ A build or build step proceeds only when it has acquired all locks. If a build or step needs a lot of locks, it may be starved [#]_ by other builds that need fewer locks. -To illustrate use of locks, a few examples. :: +To illustrate use of locks, a few examples. + +:: from buildbot import locks from buildbot.steps import source, shell @@ -113,19 +115,19 @@ To illustrate use of locks, a few examples. :: f.addStep(shell.ShellCommand(command="make all")) f.addStep(shell.ShellCommand(command="make test", locks=[db_lock.access('exclusive')])) - + b1 = {'name': 'full1', 'slavename': 'fast', 'builddir': 'f1', 'factory': f, 'locks': [build_lock.access('counting')] } - + b2 = {'name': 'full2', 'slavename': 'new', 'builddir': 'f2', 'factory': f, 'locks': [build_lock.access('counting')] } - + b3 = {'name': 'full3', 'slavename': 'old', 'builddir': 'f3', 'factory': f, 'locks': [build_lock.access('counting')] } - + b4 = {'name': 'full4', 'slavename': 'other', 'builddir': 'f4', 'factory': f, 'locks': [build_lock.access('counting')] } - + c['builders'] = [b1, b2, b3, b4] Here we have four slaves :data:`b1`, :data:`b2`, :data:`b3`, and :data:`b4`. Each @@ -158,4 +160,3 @@ preferred. situation that only a few locks are available, and they are immediately grabbed by another build. As a result, it may take a long time before all locks needed by the starved build are free at the same time. - diff --git a/master/docs/manual/cfg-properties.rst b/master/docs/manual/cfg-properties.rst index 9630c93b4fe..9ca67976936 100644 --- a/master/docs/manual/cfg-properties.rst +++ b/master/docs/manual/cfg-properties.rst @@ -149,8 +149,7 @@ parameters which do not accept properties. Property ++++++++ -The simplest form of annotation is to wrap the property name with -:class:`Property`:: +The simplest form of annotation is to wrap the property name with :class:`Property`:: from buildbot.steps.shell import ShellCommand from buildbot.process.properties import Property @@ -170,7 +169,9 @@ parameter is not set:: f.addStep(ShellCommand(command=[ 'echo', 'warnings:', Property('warnings', default='none', defaultWhenFalse=False) ])) -The default value can reference other properties, e.g., :: +The default value can reference other properties, e.g., + +:: command=Property('command', default=Property('default-command')) @@ -188,7 +189,9 @@ be interpolated into strings, instead. The tool for that job is The more common pattern is to use Python dictionary-style string interpolation by using the ``%(prop:)s`` syntax. In this form, the property name goes in the parentheses, as above. -A common mistake is to omit the trailing "s", leading to a rather obscure error from Python ("ValueError: unsupported format character"). :: +A common mistake is to omit the trailing "s", leading to a rather obscure error from Python ("ValueError: unsupported format character"). + +:: from buildbot.steps.shell import ShellCommand from buildbot.process.properties import Interpolate @@ -243,7 +246,7 @@ The following ways of interpreting the value are available. Although these are similar to shell substitutions, no other substitutions are currently supported. -Example :: +Example:: from buildbot.steps.shell import ShellCommand from buildbot.process.properties import Interpolate @@ -270,7 +273,8 @@ Renderer While Interpolate can handle many simple cases, and even some common conditionals, more complex cases are best handled with Python code. The ``renderer`` decorator creates a renderable object that will be replaced with the result of the function, called when the step it's passed to begins. -The function receives an :class:`~buildbot.interfaces.IProperties` object, which it can use to examine the values of any and all properties. For example:: +The function receives an :class:`~buildbot.interfaces.IProperties` object, which it can use to examine the values of any and all properties. +For example:: from buildbot.process import properties @properties.renderer @@ -323,17 +327,15 @@ arguments:: WithProperties("build-%s-%s.tar.gz", "branch", "revision"), "source"])) -If this :class:`BuildStep` were used in a tree obtained from Git, it would -create a tarball with a name like -:file:`build-master-a7d3a333db708e786edb34b6af646edd8d4d3ad9.tar.gz`. +If this :class:`BuildStep` were used in a tree obtained from Git, it would create a tarball with a name like :file:`build-master-a7d3a333db708e786edb34b6af646edd8d4d3ad9.tar.gz`. .. index:: unsupported format character -The more common pattern is to use Python dictionary-style string interpolation -by using the ``%(propname)s`` syntax. In this form, the property name goes in -the parentheses, as above. A common mistake is to omit the trailing "s", -leading to a rather obscure error from Python ("ValueError: unsupported format -character"). :: +The more common pattern is to use Python dictionary-style string interpolation by using the ``%(propname)s`` syntax. +In this form, the property name goes in the parentheses, as above. +A common mistake is to omit the trailing "s", leading to a rather obscure error from Python ("ValueError: unsupported format character"). + +:: from buildbot.steps.shell import ShellCommand from buildbot.process.properties import WithProperties @@ -392,7 +394,9 @@ For example:: return 'qux' ShellCommand(command=['echo', DetermineFoo()]) -or, more practically, :: +or, more practically, + +:: class Now(object): implements(IRenderable) diff --git a/master/docs/manual/cfg-schedulers.rst b/master/docs/manual/cfg-schedulers.rst index f9944b6584e..16a5c009c59 100644 --- a/master/docs/manual/cfg-schedulers.rst +++ b/master/docs/manual/cfg-schedulers.rst @@ -74,11 +74,9 @@ available with all schedulers. remembered and shown in status displays. ``codebases`` - When the scheduler processes data from more than 1 repository at the - same time then a corresponding codebase definition should be passed for each - repository. A codebase definition is a dictionary with one or more of the - following keys: repository, branch, revision. The codebase definitions have - also to be passed as dictionary. + When the scheduler processes data from more than 1 repository at the same time then a corresponding codebase definition should be passed for each repository. + A codebase definition is a dictionary with one or more of the following keys: repository, branch, revision. + The codebase definitions have also to be passed as dictionary. .. code-block:: python @@ -93,14 +91,11 @@ available with all schedulers. means that only changes with codebase '' (default value for codebase) will be accepted by the scheduler. - Buildsteps can have a reference to one of the codebases. The step will only - get information (revision, branch etc.) that is related to that codebase. - When a scheduler is triggered by new changes, these changes (having a - codebase) will be incorporated by the new build. The buildsteps referencing - to the codebases that have changes get information about those changes. - The buildstep that references to a codebase that does not have changes in - the build get the information from the codebases definition as configured in - the scheduler. + Buildsteps can have a reference to one of the codebases. + The step will only get information (revision, branch etc.) that is related to that codebase. + When a scheduler is triggered by new changes, these changes (having a codebase) will be incorporated by the new build. + The buildsteps referencing to the codebases that have changes get information about those changes. + The buildstep that references to a codebase that does not have changes in the build get the information from the codebases definition as configured in the scheduler. ``onlyImportant`` A boolean that, when ``True``, only adds important changes to the @@ -143,9 +138,9 @@ The failover is non-revertive. Change Filters ~~~~~~~~~~~~~~ -Several schedulers perform filtering on an incoming set of changes. The filter -can most generically be specified as a :class:`ChangeFilter`. Set up a -:class:`ChangeFilter` like this:: +Several schedulers perform filtering on an incoming set of changes. +The filter can most generically be specified as a :class:`ChangeFilter`. +Set up a :class:`ChangeFilter` like this:: from buildbot.changes.filter import ChangeFilter my_filter = ChangeFilter( @@ -161,7 +156,7 @@ There are five attributes of changes on which you can filter: ``project`` the project string, as defined by the ChangeSource. - + ``repository`` the repository in which this change occurred. @@ -275,7 +270,7 @@ The arguments to this scheduler are: build. If new changes are made during this interval, the timer will be restarted, so really the build will be started after a change and then after this many seconds of inactivity. - + If ``treeStableTimer`` is ``None``, then a separate build is started immediately for each Change. @@ -372,7 +367,7 @@ The arguments to this scheduler are: .. bb:sched:: Dependent .. _Dependent-Scheduler: - + Dependent Scheduler ::::::::::::::::::: @@ -435,7 +430,7 @@ Example:: .. bb:sched:: Periodic .. _Periodic-Scheduler: - + Periodic Scheduler :::::::::::::::::: @@ -569,8 +564,7 @@ again at 8:23am, but only if someone has committed code in the interim:: minute=23, onlyIfChanged=True)) -The following runs a build every two hours, using Python's :func:`range` -function:: +The following runs a build every two hours, using Python's :func:`range` function:: c.schedulers.append( timed.Nightly(name='every2hours', @@ -593,7 +587,7 @@ Finally, this example will run only on December 24th:: .. bb:sched:: Try_Userpass .. _Try-Schedulers: - + Try Schedulers :::::::::::::: @@ -640,11 +634,11 @@ currently two ways to set this up: a special file containing the source stamp information and drops it in the jobdir, just like a standard maildir. When the buildmaster notices the new file, it unpacks the information inside and starts the builds. - + The config file entries used by 'buildbot try' either specify a local queuedir (for which write and mv are used) or a remote one (using scp and ssh). - + The advantage of this scheme is that it is quite secure, the disadvantage is that it requires fiddling outside the buildmaster config (to set the permissions on the jobdir correctly). If the @@ -653,7 +647,7 @@ currently two ways to set this up: trial-build userlist. If they are on different machines, this will be much more of a hassle. It may also involve granting developer accounts on a machine that would not otherwise require them. - + To implement this, the buildslave invokes :samp:`ssh -l {username} {host} buildbot tryserver {ARGS}`, passing the patch contents over stdin. The arguments must include the inlet directory and the revision @@ -665,7 +659,7 @@ currently two ways to set this up: developer runs :command:`buildbot try`, their machine connects to the buildmaster via PB and authenticates themselves using that username and password, then sends a PB command to start the trial build. - + The advantage of this scheme is that the entire configuration is performed inside the buildmaster's config file. The disadvantages are that it is less secure (while the `cred` authentication system does @@ -789,7 +783,7 @@ Here is a fully-worked example:: builderNames=["package-all-platforms"]) c['schedulers'] = [mktarball, checkin, nightly, build, test, package] - + # on checkin, make a tarball, build it, and test it checkin_factory = factory.BuildFactory() checkin_factory.addStep(trigger.Trigger(schedulerNames=['mktarball'], diff --git a/master/docs/manual/cfg-statustargets.rst b/master/docs/manual/cfg-statustargets.rst index 8bffa6267c1..ce377b2b951 100644 --- a/master/docs/manual/cfg-statustargets.rst +++ b/master/docs/manual/cfg-statustargets.rst @@ -48,260 +48,6 @@ show status for Builders that are in one of the named categories. The remainder of this section describes each built-in status target. A full list of status targets is available in the :bb:index:`status`. -.. _Change-Hooks: - -Change Hooks -~~~~~~~~~~~~ - -.. warning:: - - Tihs section corresponds to the WebStatus, which has been removed. - The content remains here for a later move to another location. - -The ``/change_hook`` url is a magic URL which will accept HTTP requests and translate -them into changes for buildbot. Implementations (such as a trivial json-based endpoint -and a GitHub implementation) can be found in :bb:src:`master/buildbot/status/web/hooks`. -The format of the url is :samp:`/change_hook/{DIALECT}` where DIALECT is a package within the -hooks directory. Change_hook is disabled by default and each DIALECT has to be enabled -separately, for security reasons - -An example WebStatus configuration line which enables change_hook and two DIALECTS:: - - c['status'].append(html.WebStatus(http_port=8010,allowForce=True, - change_hook_dialects={ - 'base': True, - 'somehook': {'option1':True, - 'option2':False}})) - -Within the WebStatus arguments, the ``change_hook`` key enables/disables the module -and ``change_hook_dialects`` whitelists DIALECTs where the keys are the module names -and the values are optional arguments which will be passed to the hooks. - -The :file:`post_build_request.py` script in :file:`master/contrib` allows for the -submission of an arbitrary change request. Run :command:`post_build_request.py ---help` for more information. The ``base`` dialect must be enabled for this to -work. - -GitHub hook -+++++++++++ - -The GitHub hook is simple and takes no options. :: - - c['status'].append(html.WebStatus(..., - change_hook_dialects={ 'github' : True })) - -With this set up, add a Post-Receive URL for the project in the GitHub -administrative interface, pointing to ``/change_hook/github`` relative to -the root of the web status. For example, if the grid URL is -``http://builds.mycompany.com/bbot/grid``, then point GitHub to -``http://builds.mycompany.com/bbot/change_hook/github``. To specify a project -associated to the repository, append ``?project=name`` to the URL. - -Note that there is a standalone HTTP server available for receiving GitHub -notifications, as well: :file:`contrib/github_buildbot.py`. This script may be -useful in cases where you cannot expose the WebStatus for public consumption. - -.. warning:: - - The incoming HTTP requests for this hook are not authenticated by default. - Anyone who can access the web status can "fake" a request from - GitHub, potentially causing the buildmaster to run arbitrary code. - -To protect URL against unauthorized access you should use ``change_hook_auth`` option :: - - c['status'].append(html.WebStatus(..., - change_hook_auth=["file:changehook.passwd"])) - -And create a file ``changehook.passwd`` - -.. code-block:: none - - user:password - -Then, create a GitHub service hook (see https://help.github.com/articles/post-receive-hooks) with a WebHook URL like ``http://user:password@builds.mycompany.com/bbot/change_hook/github``. - -See the `documentation `_ for twisted cred for more option to pass to ``change_hook_auth``. - -Note that not using ``change_hook_auth`` can expose you to security risks. - -BitBucket hook -++++++++++++++ - -The BitBucket hook is as simple as GitHub one and it also takes no options. :: - - c['status'].append(html.WebStatus(..., - change_hook_dialects={ 'bitbucket' : True })) - -When this is setup you should add a `POST` service pointing to ``/change_hook/bitbucket`` -relative to the root of the web status. For example, it the grid URL is -``http://builds.mycompany.com/bbot/grid``, then point BitBucket to -``http://builds.mycompany.com/change_hook/bitbucket``. To specify a project associated -to the repository, append ``?project=name`` to the URL. - -Note that there is a satandalone HTTP server available for receiving BitBucket -notifications, as well: :file:`contrib/bitbucket_buildbot.py`. This script may be -useful in cases where you cannot expose the WebStatus for public consumption. - -.. warning:: - - As in the previous case, the incoming HTTP requests for this hook are not - authenticated bu default. Anyone who can access the web status can "fake" - a request from BitBucket, potentially causing the buildmaster to run - arbitrary code. - -To protect URL against unauthorized access you should use ``change_hook_auth`` option. :: - - c['status'].append(html.WebStatus(..., - change_hook_auth=["file:changehook.passwd"])) - -Then, create a BitBucket service hook (see https://confluence.atlassian.com/display/BITBUCKET/POST+Service+Management) with a WebHook URL like ``http://user:password@builds.mycompany.com/bbot/change_hook/bitbucket``. - -Note that as before, not using ``change_hook_auth`` can expose you to security risks. - -Google Code hook -++++++++++++++++ - -The Google Code hook is quite similar to the GitHub Hook. It has one option -for the "Post-Commit Authentication Key" used to check if the request is -legitimate:: - - c['status'].append(html.WebStatus( - # ... - change_hook_dialects={'googlecode': {'secret_key': 'FSP3p-Ghdn4T0oqX'}} - )) - -This will add a "Post-Commit URL" for the project in the Google Code -administrative interface, pointing to ``/change_hook/googlecode`` relative to -the root of the web status. - -Alternatively, you can use the :ref:`GoogleCodeAtomPoller` :class:`ChangeSource` -that periodically poll the Google Code commit feed for changes. - -.. note:: - - Google Code doesn't send the branch on which the changes were made. So, the - hook always returns ``'default'`` as the branch, you can override it with the - ``'branch'`` option:: - - change_hook_dialects={'googlecode': {'secret_key': 'FSP3p-Ghdn4T0oqX', 'branch': 'master'}} - -Poller hook -+++++++++++ - -The poller hook allows you to use GET or POST requests to trigger -polling. One advantage of this is your buildbot instance can poll -at launch (using the pollAtLaunch flag) to get changes that happened -while it was down, but then you can still use a commit hook to get -fast notification of new changes. - -Suppose you have a poller configured like this:: - - c['change_source'] = SVNPoller( - svnurl="https://amanda.svn.sourceforge.net/svnroot/amanda/amanda", - split_file=split_file_branches, - pollInterval=24*60*60, - pollAtLaunch=True) - -And you configure your WebStatus to enable this hook:: - - c['status'].append(html.WebStatus( - # ... - change_hook_dialects={'poller': True} - )) - -Then you will be able to trigger a poll of the SVN repository by poking the -``/change_hook/poller`` URL from a commit hook like this: - -.. code-block:: bash - - curl -s -F poller=https://amanda.svn.sourceforge.net/svnroot/amanda/amanda \ - http://yourbuildbot/change_hook/poller - -If no ``poller`` argument is provided then the hook will trigger polling of all -polling change sources. - -You can restrict which pollers the webhook has access to using the ``allowed`` -option:: - - c['status'].append(html.WebStatus( - # ... - change_hook_dialects={'poller': {'allowed': ['https://amanda.svn.sourceforge.net/svnroot/amanda/amanda']}} - )) - -GitLab hook -+++++++++++ - -The GitLab hook is as simple as GitHub one and it also takes no options. :: - - c['status'].append(html.WebStatus( - # ... - change_hook_dialects={ 'gitlab' : True } - )) - -When this is setup you should add a `POST` service pointing to ``/change_hook/gitlab`` -relative to the root of the web status. For example, it the grid URL is -``http://builds.mycompany.com/bbot/grid``, then point GitLab to -``http://builds.mycompany.com/change_hook/gitlab``. The project and/or codebase can -also be passed in the URL by appending ``?project=name`` or ``?codebase=foo`` to the URL. -These parameters will be passed along to the scheduler. - -.. warning:: - - As in the previous case, the incoming HTTP requests for this hook are not - authenticated bu default. Anyone who can access the web status can "fake" - a request from your GitLab server, potentially causing the buildmaster to run - arbitrary code. - -To protect URL against unauthorized access you should use ``change_hook_auth`` option. :: - - c['status'].append(html.WebStatus( - # ... - change_hook_auth=["file:changehook.passwd"] - )) - -Then, create a GitLab service hook (see https://your.gitlab.server/help/web_hooks) with a WebHook URL like ``http://user:password@builds.mycompany.com/bbot/change_hook/gitlab``. - -Note that as before, not using ``change_hook_auth`` can expose you to security risks. - -Gitorious Hook -++++++++++++++ - -The Gitorious hook is as simple as GitHub one and it also takes no options. :: - - c['status'].append(html.WebStatus( - # ... - change_hook_dialects={'gitorious': True} - )) - -When this is setup you should add a `POST` service pointing to ``/change_hook/gitorious`` -relative to the root of the web status. For example, it the grid URL is -``http://builds.mycompany.com/bbot/grid``, then point Gitorious to -``http://builds.mycompany.com/change_hook/gitorious``. - -.. warning:: - - As in the previous case, the incoming HTTP requests for this hook are not - authenticated by default. Anyone who can access the web status can "fake" - a request from your Gitorious server, potentially causing the buildmaster to run - arbitrary code. - -To protect URL against unauthorized access you should use ``change_hook_auth`` option. :: - - c['status'].append(html.WebStatus( - # ... - change_hook_auth=["file:changehook.passwd"] - )) - -Then, create a Gitorious web hook (see http://gitorious.org/gitorious/pages/WebHooks) with a WebHook URL like ``http://user:password@builds.mycompany.com/bbot/change_hook/gitorious``. - -Note that as before, not using ``change_hook_auth`` can expose you to security risks. - -.. note:: - - Web hooks are only available for local Gitorious - installations, since this feature is not offered as part of - Gitorious.org yet. - .. bb:status:: MailNotifier .. index:: single: email; MailNotifier @@ -344,7 +90,9 @@ different kinds of mail to different recipients, use multiple The following simple example will send an email upon the completion of each build, to just those developers whose :class:`Change`\s were included in the build. The email contains a description of the :class:`Build`, its results, -and URLs where more information can be obtained. :: +and URLs where more information can be obtained. + +:: from buildbot.status.mail import MailNotifier mn = MailNotifier(fromaddr="buildbot@example.org", lookup="example.org") @@ -387,7 +135,7 @@ provided in a standard MailNotifier message. For this purpose MailNotifier provides the argument ``messageFormatter`` (a function) which allows for the creation of messages with unique content. -For example, if only short emails are desired (e.g., for delivery to phones) :: +For example, if only short emails are desired (e.g., for delivery to phones):: from buildbot.status.builder import Results def messageFormatter(mode, name, build, results, master_status): @@ -675,7 +423,7 @@ MailNotifier mode ``mode`` (a combination of ``change``, ``failing``, ``passing``, ``problem``, ``warnings``, ``exception``, ``all``) -Builder result as a string :: +Builder result as a string:: from buildbot.status.builder import Results result_str = Results[results] @@ -736,7 +484,7 @@ Source information (only valid if ss is not ``None``) information above. :meth:`asText` returns a list of strings and :meth:`asDict` returns a dictionary suitable for html/mail rendering. -Log information :: +Log information:: logs = list() for log in build.getLogs(): @@ -759,10 +507,10 @@ IRC Bot .. py:class:: buildbot.status.words.IRC -The :class:`buildbot.status.words.IRC` status target creates an IRC bot -which will attach to certain channels and be available for status -queries. It can also be asked to announce builds as they occur, or be -told to shut up. :: +The :class:`buildbot.status.words.IRC` status target creates an IRC bot which will attach to certain channels and be available for status queries. +It can also be asked to announce builds as they occur, or be told to shut up. + +:: from buildbot.status import words irc = words.IRC("irc.example.org", "botnickname", @@ -1122,6 +870,260 @@ empty string, the plugin will skip sending the status. You can define custom start and end build messages using the `startDescription` and `endDescription` optional interpolation arguments. +.. _Change-Hooks: + +Change Hooks +~~~~~~~~~~~~ + +.. warning:: + + Tihs section corresponds to the WebStatus, which has been removed. + The content remains here for a later move to another location. + +The ``/change_hook`` url is a magic URL which will accept HTTP requests and translate +them into changes for buildbot. Implementations (such as a trivial json-based endpoint +and a GitHub implementation) can be found in :bb:src:`master/buildbot/status/web/hooks`. +The format of the url is :samp:`/change_hook/{DIALECT}` where DIALECT is a package within the +hooks directory. Change_hook is disabled by default and each DIALECT has to be enabled +separately, for security reasons + +An example WebStatus configuration line which enables change_hook and two DIALECTS:: + + c['status'].append(html.WebStatus(http_port=8010,allowForce=True, + change_hook_dialects={ + 'base': True, + 'somehook': {'option1':True, + 'option2':False}})) + +Within the WebStatus arguments, the ``change_hook`` key enables/disables the module +and ``change_hook_dialects`` whitelists DIALECTs where the keys are the module names +and the values are optional arguments which will be passed to the hooks. + +The :file:`post_build_request.py` script in :file:`master/contrib` allows for the +submission of an arbitrary change request. Run :command:`post_build_request.py +--help` for more information. The ``base`` dialect must be enabled for this to +work. + +GitHub hook ++++++++++++ + +The GitHub hook is simple and takes no options. :: + + c['status'].append(html.WebStatus(..., + change_hook_dialects={ 'github' : True })) + +With this set up, add a Post-Receive URL for the project in the GitHub +administrative interface, pointing to ``/change_hook/github`` relative to +the root of the web status. For example, if the grid URL is +``http://builds.mycompany.com/bbot/grid``, then point GitHub to +``http://builds.mycompany.com/bbot/change_hook/github``. To specify a project +associated to the repository, append ``?project=name`` to the URL. + +Note that there is a standalone HTTP server available for receiving GitHub +notifications, as well: :file:`contrib/github_buildbot.py`. This script may be +useful in cases where you cannot expose the WebStatus for public consumption. + +.. warning:: + + The incoming HTTP requests for this hook are not authenticated by default. + Anyone who can access the web status can "fake" a request from + GitHub, potentially causing the buildmaster to run arbitrary code. + +To protect URL against unauthorized access you should use ``change_hook_auth`` option :: + + c['status'].append(html.WebStatus(..., + change_hook_auth=["file:changehook.passwd"])) + +And create a file ``changehook.passwd`` + +.. code-block:: none + + user:password + +Then, create a GitHub service hook (see https://help.github.com/articles/post-receive-hooks) with a WebHook URL like ``http://user:password@builds.mycompany.com/bbot/change_hook/github``. + +See the `documentation `_ for twisted cred for more option to pass to ``change_hook_auth``. + +Note that not using ``change_hook_auth`` can expose you to security risks. + +BitBucket hook +++++++++++++++ + +The BitBucket hook is as simple as GitHub one and it also takes no options. :: + + c['status'].append(html.WebStatus(..., + change_hook_dialects={ 'bitbucket' : True })) + +When this is setup you should add a `POST` service pointing to ``/change_hook/bitbucket`` +relative to the root of the web status. For example, it the grid URL is +``http://builds.mycompany.com/bbot/grid``, then point BitBucket to +``http://builds.mycompany.com/change_hook/bitbucket``. To specify a project associated +to the repository, append ``?project=name`` to the URL. + +Note that there is a satandalone HTTP server available for receiving BitBucket +notifications, as well: :file:`contrib/bitbucket_buildbot.py`. This script may be +useful in cases where you cannot expose the WebStatus for public consumption. + +.. warning:: + + As in the previous case, the incoming HTTP requests for this hook are not + authenticated bu default. Anyone who can access the web status can "fake" + a request from BitBucket, potentially causing the buildmaster to run + arbitrary code. + +To protect URL against unauthorized access you should use ``change_hook_auth`` option. :: + + c['status'].append(html.WebStatus(..., + change_hook_auth=["file:changehook.passwd"])) + +Then, create a BitBucket service hook (see https://confluence.atlassian.com/display/BITBUCKET/POST+Service+Management) with a WebHook URL like ``http://user:password@builds.mycompany.com/bbot/change_hook/bitbucket``. + +Note that as before, not using ``change_hook_auth`` can expose you to security risks. + +Google Code hook +++++++++++++++++ + +The Google Code hook is quite similar to the GitHub Hook. It has one option +for the "Post-Commit Authentication Key" used to check if the request is +legitimate:: + + c['status'].append(html.WebStatus( + # ... + change_hook_dialects={'googlecode': {'secret_key': 'FSP3p-Ghdn4T0oqX'}} + )) + +This will add a "Post-Commit URL" for the project in the Google Code +administrative interface, pointing to ``/change_hook/googlecode`` relative to +the root of the web status. + +Alternatively, you can use the :ref:`GoogleCodeAtomPoller` :class:`ChangeSource` +that periodically poll the Google Code commit feed for changes. + +.. note:: + + Google Code doesn't send the branch on which the changes were made. So, the + hook always returns ``'default'`` as the branch, you can override it with the + ``'branch'`` option:: + + change_hook_dialects={'googlecode': {'secret_key': 'FSP3p-Ghdn4T0oqX', 'branch': 'master'}} + +Poller hook ++++++++++++ + +The poller hook allows you to use GET or POST requests to trigger +polling. One advantage of this is your buildbot instance can poll +at launch (using the pollAtLaunch flag) to get changes that happened +while it was down, but then you can still use a commit hook to get +fast notification of new changes. + +Suppose you have a poller configured like this:: + + c['change_source'] = SVNPoller( + svnurl="https://amanda.svn.sourceforge.net/svnroot/amanda/amanda", + split_file=split_file_branches, + pollInterval=24*60*60, + pollAtLaunch=True) + +And you configure your WebStatus to enable this hook:: + + c['status'].append(html.WebStatus( + # ... + change_hook_dialects={'poller': True} + )) + +Then you will be able to trigger a poll of the SVN repository by poking the +``/change_hook/poller`` URL from a commit hook like this: + +.. code-block:: bash + + curl -s -F poller=https://amanda.svn.sourceforge.net/svnroot/amanda/amanda \ + http://yourbuildbot/change_hook/poller + +If no ``poller`` argument is provided then the hook will trigger polling of all +polling change sources. + +You can restrict which pollers the webhook has access to using the ``allowed`` +option:: + + c['status'].append(html.WebStatus( + # ... + change_hook_dialects={'poller': {'allowed': ['https://amanda.svn.sourceforge.net/svnroot/amanda/amanda']}} + )) + +GitLab hook ++++++++++++ + +The GitLab hook is as simple as GitHub one and it also takes no options. :: + + c['status'].append(html.WebStatus( + # ... + change_hook_dialects={ 'gitlab' : True } + )) + +When this is setup you should add a `POST` service pointing to ``/change_hook/gitlab`` +relative to the root of the web status. For example, it the grid URL is +``http://builds.mycompany.com/bbot/grid``, then point GitLab to +``http://builds.mycompany.com/change_hook/gitlab``. The project and/or codebase can +also be passed in the URL by appending ``?project=name`` or ``?codebase=foo`` to the URL. +These parameters will be passed along to the scheduler. + +.. warning:: + + As in the previous case, the incoming HTTP requests for this hook are not + authenticated bu default. Anyone who can access the web status can "fake" + a request from your GitLab server, potentially causing the buildmaster to run + arbitrary code. + +To protect URL against unauthorized access you should use ``change_hook_auth`` option. :: + + c['status'].append(html.WebStatus( + # ... + change_hook_auth=["file:changehook.passwd"] + )) + +Then, create a GitLab service hook (see https://your.gitlab.server/help/web_hooks) with a WebHook URL like ``http://user:password@builds.mycompany.com/bbot/change_hook/gitlab``. + +Note that as before, not using ``change_hook_auth`` can expose you to security risks. + +Gitorious Hook +++++++++++++++ + +The Gitorious hook is as simple as GitHub one and it also takes no options. :: + + c['status'].append(html.WebStatus( + # ... + change_hook_dialects={'gitorious': True} + )) + +When this is setup you should add a `POST` service pointing to ``/change_hook/gitorious`` +relative to the root of the web status. For example, it the grid URL is +``http://builds.mycompany.com/bbot/grid``, then point Gitorious to +``http://builds.mycompany.com/change_hook/gitorious``. + +.. warning:: + + As in the previous case, the incoming HTTP requests for this hook are not + authenticated by default. Anyone who can access the web status can "fake" + a request from your Gitorious server, potentially causing the buildmaster to run + arbitrary code. + +To protect URL against unauthorized access you should use ``change_hook_auth`` option. :: + + c['status'].append(html.WebStatus( + # ... + change_hook_auth=["file:changehook.passwd"] + )) + +Then, create a Gitorious web hook (see http://gitorious.org/gitorious/pages/WebHooks) with a WebHook URL like ``http://user:password@builds.mycompany.com/bbot/change_hook/gitorious``. + +Note that as before, not using ``change_hook_auth`` can expose you to security risks. + +.. note:: + + Web hooks are only available for local Gitorious + installations, since this feature is not offered as part of + Gitorious.org yet. + .. [#] Apparently this is the same way http://buildd.debian.org displays build status @@ -1130,4 +1132,3 @@ You can define custom start and end build messages using the but this is completely untested .. _PyOpenSSL: http://pyopenssl.sourceforge.net/ - diff --git a/master/docs/manual/cfg-www.rst b/master/docs/manual/cfg-www.rst index 01a0321ee5e..6b9545a406d 100644 --- a/master/docs/manual/cfg-www.rst +++ b/master/docs/manual/cfg-www.rst @@ -39,7 +39,7 @@ This server is configured with the :bb:cfg:`www` configuration key, which specif ``plugins`` This key gives a dictionary of additional UI plugins to load, along with configuration for those plugins. These plugins must be separately installed in the Python environment, e.g., ``pip install buildbot-www-waterfall``. - For example :: + For example:: c['www'] = { 'plugins': {'waterfall': {'num_builds': 50}} @@ -91,7 +91,9 @@ The available classes are described here: :param users: list of ``("user","password")`` tuples, or a dictionary of ``{"user": "password", ..}`` - Simple username/password authentication using a list of user/password tuples provided in the configuration file. :: + Simple username/password authentication using a list of user/password tuples provided in the configuration file. + + :: from buildbot.www.auth import UserPasswordAuth c['www'] = { @@ -101,9 +103,11 @@ The available classes are described here: .. py:class:: buildbot.www.auth.HTPasswdAuth(passwdFile) - :param passwdFile: An ``.htpasswd`` file to read + :param passwdFile: An :file:`.htpasswd` file to read + + This class implements simple username/password authentication against a standard :file:`.htpasswd` file. - This class implements simple username/password authentication against a standard :file:`.htpasswd` file. :: + :: from buildbot.www.auth import HTPasswdAuth c['www'] = { @@ -272,6 +276,3 @@ Currently only one provider is available: .. code-block:: bash pip install python3-ldap - - - diff --git a/master/docs/manual/customization.rst b/master/docs/manual/customization.rst index e94920b052a..d2080a8b242 100644 --- a/master/docs/manual/customization.rst +++ b/master/docs/manual/customization.rst @@ -1,5 +1,3 @@ -.. _Customization: - Customization ============= @@ -16,14 +14,14 @@ Programmatic Configuration Generation Bearing in mind that ``master.cfg`` is a Python file, large configurations can be shortened considerably by judicious use of Python loops. For example, the following will generate a builder for each of a range of supported versions of Python:: - pythons = [ 'python2.4', 'python2.5', 'python2.6', 'python2.7', - 'python3.2', 'python3.3' ] - pytest_slaves = [ "slave%s" % n for n in range(10) ] + pythons = ['python2.4', 'python2.5', 'python2.6', 'python2.7', + 'python3.2', 'python3.3'] + pytest_slaves = ["slave%s" % n for n in range(10)] for python in pythons: - f = BuildFactory() - f.addStep(SVN(...)) - f.addStep(ShellCommand(command=[ python, 'test.py' ])) - c['builders'].append(BuilderConfig( + f = util.BuildFactory() + f.addStep(steps.SVN(...)) + f.addStep(steps.ShellCommand(command=[python, 'test.py'])) + c['builders'].append(util.BuilderConfig( name="test-%s" % python, factory=f, slavenames=pytest_slaves)) @@ -53,11 +51,13 @@ For example:: In many cases, the details of the :class:`SourceStamp`\s and :class:`BuildRequest`\s are important. In this example, only :class:`BuildRequest`\s with the same "reason" are merged; thus developers forcing builds for different reasons will see distinct builds. Note the use of the :func:`canBeMergedWith` method to access the source stamp compatibility algorithm. -Note, in particular, that this function returns a Deferred as of Buildbot-0.9.0. :: +Note, in particular, that this function returns a Deferred as of Buildbot-0.9.0. + +:: @defer.inlineCallbacks def mergeRequests(builder, req1, req2): - if (yield req1.source.canBeMergedWith(req2.source)) and req1.reason == req2.reason: + if (yield req1.source.canBeMergedWith(req2.source)) and req1.reason == req2.reason: defer.returnValue(True) else: defer.returnValue(False) @@ -134,20 +134,22 @@ If some non-immediate result must be calculated, the ``nextBuild`` function can d.addCallback(pick) return d -The ``nextBuild`` function is passed as parameter to :class:`BuilderConfig`. +The ``nextBuild`` function is passed as parameter to :class:`BuilderConfig`:: + + ... BuilderConfig(..., nextBuild=nextBuild, ...) ... .. _Customizing-SVNPoller: Customizing SVNPoller --------------------- -Each source file that is tracked by a Subversion repository has a fully-qualified SVN URL in the following form: ``({REPOURL})({PROJECT-plus-BRANCH})({FILEPATH})``. -When you create the :bb:chsrc:`SVNPoller`, you give it a ``svnurl`` value that includes all of the ``{REPOURL}`` and possibly some portion of the ``{PROJECT-plus-BRANCH}`` string. -The :bb:chsrc:`SVNPoller` is responsible for producing Changes that contain a branch name and a ``{FILEPATH}`` (which is relative to the top of a checked-out tree). +Each source file that is tracked by a Subversion repository has a fully-qualified SVN URL in the following form: :samp:`({REPOURL})({PROJECT-plus-BRANCH})({FILEPATH})`. +When you create the :bb:chsrc:`SVNPoller`, you give it a ``svnurl`` value that includes all of the :samp:`{REPOURL}` and possibly some portion of the :samp:`{PROJECT-plus-BRANCH}` string. +The :bb:chsrc:`SVNPoller` is responsible for producing Changes that contain a branch name and a :samp:`{FILEPATH}` (which is relative to the top of a checked-out tree). The details of how these strings are split up depend upon how your repository names its branches. -PROJECT/BRANCHNAME/FILEPATH repositories -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +:samp:`{PROJECT}/{BRANCHNAME}/{FILEPATH}` repositories +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ One common layout is to have all the various projects that share a repository get a single top-level directory each, with ``branches``, ``tags``, and ``trunk`` subdirectories: @@ -235,8 +237,8 @@ Note here that we are monitoring at the root of the repository, and that within It is that ``amanda`` subdirectory whose name becomes the ``project`` field of the Change. -BRANCHNAME/PROJECT/FILEPATH repositories -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +:samp:`{BRANCHNAME}/{PROJECT}/{FILEPATH}` repositories +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Another common way to organize a Subversion repository is to put the branch name at the top, and the projects underneath. This is especially frequent when there are a number of related sub-projects that all get released in a group. @@ -904,7 +906,10 @@ The :class:`BuildStep` class definition itself will look something like this:: So that's the code that we want to wind up using. How do we actually deploy it? -You have a couple of different options. +You have a number of different options: + +.. contents:: + :local: Inclusion in the :file:`master.cfg` file ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ @@ -915,7 +920,7 @@ The simplest technique is to simply put the step class definitions in your :file f.addStep(SVN(svnurl="stuff")) f.addStep(Framboozle()) -Remember that :file:`master.cfg` is secretly just a Python program with one job: populating the :file:`BuildmasterConfig` dictionary. +Remember that :file:`master.cfg` is secretly just a Python program with one job: populating the :data:`BuildmasterConfig` dictionary. And Python programs are allowed to define as many classes as they like. So you can define classes and use them in the same file, just as long as the class is defined before some other code tries to use it. @@ -1011,11 +1016,14 @@ In either case, post a note about your patch to the mailing list, so others can f.addStep(SVN(svnurl="stuff")) f.addStep(framboozle.Framboozle()) -And then you don't even have to install framboozle.py anywhere on your system, since it will ship with Buildbot. +And then you don't even have to install :file:`framboozle.py` anywhere on your system, since it will ship with Buildbot. You don't have to be root, you don't have to set :envvar:`PYTHONPATH`. But you do have to make a good case for Framboozle being worth going into the main distribution, you'll probably have to provide docs and some unit test cases, you'll need to figure out what kind of beer the author likes (IPA's and Stouts for Dustin), and then you'll have to wait until the next release. But in some environments, all this is easier than getting root on your buildmaster box, so the tradeoffs may actually be worth it. +Summary +~~~~~~~ + Putting the code in master.cfg (1) makes it available to that buildmaster instance. Putting it in a file in a personal library directory (2) makes it available for any buildmasters you might be running. Putting it in a file in a system-wide shared library directory (3) makes it available for any buildmasters that anyone on that system might be running. diff --git a/master/docs/tutorial/fiveminutes.rst b/master/docs/tutorial/fiveminutes.rst index 4ee6e5dca33..15c6a282e57 100644 --- a/master/docs/tutorial/fiveminutes.rst +++ b/master/docs/tutorial/fiveminutes.rst @@ -352,7 +352,7 @@ to not hardcode all the schedulers and builders in the configuration, but generate them dynamically starting from list of all projects, branches, targets etc. and using loops to generate all possible combinations (or only the needed ones, depending on the specific setup), as explained in -the documentation chapter about :ref:`Customization`. +the documentation chapter about :doc:`../manual/customization`. Status targets --------------