.. index:: File Transfer
.. bb:step:: FileUpload
.. bb:step:: FileDownload
.. py:class:: buildbot.steps.transfer.FileUpload
.. py:class:: buildbot.steps.transfer.FileDownload
Most of the work involved in a build will take place on the worker. But occasionally it is useful to do some work on the buildmaster side. The most basic way to involve the buildmaster is simply to move a file from the worker to the master, or vice versa. There are a pair of steps named :bb:step:`FileUpload` and :bb:step:`FileDownload` to provide this functionality. :bb:step:`FileUpload` moves a file up to the master, while :bb:step:`FileDownload` moves a file down from the master.
As an example, let's assume that there is a step which produces an HTML file within the source tree that contains some sort of generated project documentation. And let's assume that we run nginx web server on the buildmaster host for serving static files. We want to move this file to the buildmaster, into a :file:`/usr/share/nginx/www/` directory, so it can be visible to developers. This file will wind up in the worker-side working directory under the name :file:`docs/reference.html`. We want to put it into the master-side :file:`/usr/share/nginx/www/ref.html`, and add a link to the HTML status to the uploaded file.
from buildbot.plugins import steps
f.addStep(steps.ShellCommand(command=["make", "docs"]))
f.addStep(steps.FileUpload(workersrc="docs/reference.html",
masterdest="/usr/share/nginx/www/ref.html",
url="http://somesite/~buildbot/ref.html"))The masterdest= argument will be passed to :meth:`os.path.expanduser`, so things like ~ will be expanded properly.
Non-absolute paths will be interpreted relative to the buildmaster's base directory.
Likewise, the workersrc= argument will be expanded and interpreted relative to the builder's working directory.
Note
The copied file will have the same permissions on the master as on the worker, look at the mode= parameter to set it differently.
To move a file from the master to the worker, use the :bb:step:`FileDownload` command. For example, let's assume that some step requires a configuration file that, for whatever reason, could not be recorded in the source code repository or generated on the worker side:
from buildbot.plugins import steps
f.addStep(steps.FileDownload(mastersrc="~/todays_build_config.txt",
workerdest="build_config.txt"))
f.addStep(steps.ShellCommand(command=["make", "config"]))Like :bb:step:`FileUpload`, the mastersrc= argument is interpreted relative to the buildmaster's base directory, and the workerdest= argument is relative to the builder's working directory.
If the worker is running in :file:`~worker`, and the builder's builddir is something like :file:`tests-i386`, then the workdir is going to be :file:`~worker/tests-i386/build`, and a workerdest= of :file:`foo/bar.html` will get put in :file:`~worker/tests-i386/build/foo/bar.html`.
Both of these commands will create any missing intervening directories.
The maxsize= argument lets you set a maximum size for the file to be transferred.
This may help to avoid surprises: transferring a 100MB coredump when you were expecting to move a 10kB status file might take an awfully long time.
The blocksize= argument controls how the file is sent over the network: larger blocksizes are slightly more efficient but also consume more memory on each end, and there is a hard-coded limit of about 640kB.
The mode= argument allows you to control the access permissions of the target file, traditionally expressed as an octal integer.
The most common value is probably 0o755, which sets the x executable bit on the file (useful for shell scripts and the like).
The default value for mode= is None, which means the permission bits will default to whatever the umask of the writing process is.
The default umask tends to be fairly restrictive, but at least on the worker you can make it less restrictive with a --umask command-line option at creation time (:ref:`Worker-Options`).
The keepstamp= argument is a boolean that, when True, forces the modified and accessed time of the destination file to match the times of the source file.
When False (the default), the modified and accessed times of the destination file are set to the current time on the buildmaster.
The url= argument allows you to specify an url that will be displayed in the HTML status.
The title of the url will be the name of the item transferred (directory for :class:`DirectoryUpload` or file for :class:`FileUpload`).
This allows the user to add a link to the uploaded item if that one is uploaded to an accessible place.
For :bb:step:`FileUpload`, the urlText= argument allows you to specify the url title that will be displayed in the web UI.
.. bb:step:: DirectoryUpload
.. py:class:: buildbot.steps.transfer.DirectoryUpload
To transfer complete directories from the worker to the master, there is a :class:`BuildStep` named :bb:step:`DirectoryUpload`.
It works like :bb:step:`FileUpload`, just for directories.
However it does not support the maxsize, blocksize and mode arguments.
As an example, let's assume an generated project documentation, which consists of many files (like the output of :command:`doxygen` or :command:`epydoc`).
And let's assume that we run nginx web server on buildmaster host for serving static files.
We want to move the entire documentation to the buildmaster, into a :file:`/usr/share/nginx/www/docs` directory, and add a link to the uploaded documentation on the HTML status page.
On the worker-side the directory can be found under :file:`docs`:
from buildbot.plugins import steps
f.addStep(steps.ShellCommand(command=["make", "docs"]))
f.addStep(steps.DirectoryUpload(workersrc="docs",
masterdest="/usr/share/nginx/www/docs",
url="~buildbot/docs"))The :bb:step:`DirectoryUpload` step will create all necessary directories and transfers empty directories, too.
The maxsize and blocksize parameters are the same as for :bb:step:`FileUpload`, although note that the size of the transferred data is implementation-dependent, and probably much larger than you expect due to the encoding used (currently tar).
The optional compress argument can be given as 'gz' or 'bz2' to compress the datastream.
For :bb:step:`DirectoryUpload` the urlText= argument allows you to specify the url title that will be displayed in the web UI.
Note
The permissions on the copied files will be the same on the master as originally on the worker, see option buildbot-worker create-worker --umask to change the default one.
.. bb:step:: MultipleFileUpload
.. py:class:: buildbot.steps.transfer.MultipleFileUpload
In addition to the :bb:step:`FileUpload` and :bb:step:`DirectoryUpload` steps there is the :bb:step:`MultipleFileUpload` step for uploading a bunch of files (and directories) in a single :class:`BuildStep`.
The step supports all arguments that are supported by :bb:step:`FileUpload` and :bb:step:`DirectoryUpload`, but instead of a the single workersrc parameter it takes a (plural) workersrcs parameter.
This parameter should either be a list, something that can be rendered as a list or a string which will be converted to a list.
Additionally it supports the glob parameter if this parameter is set to True all arguments in workersrcs will be parsed through glob and the results will be uploaded to masterdest.:
from buildbot.plugins import steps
f.addStep(steps.ShellCommand(command=["make", "test"]))
f.addStep(steps.ShellCommand(command=["make", "docs"]))
f.addStep(steps.MultipleFileUpload(workersrcs=["docs", "test-results.html"],
masterdest="/usr/share/nginx/www/",
url="~buildbot"))The url= parameter, can be used to specify a link to be displayed in the HTML status of the step.
The way URLs are added to the step can be customized by extending the :bb:step:`MultipleFileUpload` class. The allUploadsDone method is called after all files have been uploaded and sets the URL. The uploadDone method is called once for each uploaded file and can be used to create file-specific links.
import os
from buildbot.plugins import steps
class CustomFileUpload(steps.MultipleFileUpload):
linkTypes = ('.html', '.txt')
def linkFile(self, basename):
name, ext = os.path.splitext(basename)
return ext in self.linkTypes
def uploadDone(self, result, source, masterdest):
if self.url:
basename = os.path.basename(source)
if self.linkFile(basename):
self.addURL(self.url + '/' + basename, basename)
def allUploadsDone(self, result, sources, masterdest):
if self.url:
notLinked = [src for src in sources if not self.linkFile(src)]
numFiles = len(notLinked)
if numFiles:
self.addURL(self.url, '... %d more' % numFiles)For :bb:step:`MultipleFileUpload` the urlText= argument allows you to specify the url title that will be displayed in the web UI.
.. bb:step:: StringDownload
.. bb:step:: JSONStringDownload
.. bb:step:: JSONPropertiesDownload
.. py:class:: buildbot.steps.transfer.StringDownload
.. py:class:: buildbot.steps.transfer.JSONStringDownload
.. py:class:: buildbot.steps.transfer.JSONPropertiesDownload
Sometimes it is useful to transfer a calculated value from the master to the worker. Instead of having to create a temporary file and then use FileDownload, you can use one of the string download steps.
from buildbot.plugins import steps, util
f.addStep(steps.StringDownload(util.Interpolate("%(src::branch)s-%(prop:got_revision)s\n"),
workerdest="buildid.txt")):bb:step:`StringDownload` works just like :bb:step:`FileDownload` except it takes a single argument, s, representing the string to download instead of a mastersrc argument.
from buildbot.plugins import steps
buildinfo = {
'branch': Property('branch'),
'got_revision': Property('got_revision')
}
f.addStep(steps.JSONStringDownload(buildinfo, workerdest="buildinfo.json")):bb:step:`JSONStringDownload` is similar, except it takes an o argument, which must be JSON serializable, and transfers that as a JSON-encoded string to the worker.
.. index:: Properties; JSONPropertiesDownload
from buildbot.plugins import steps
f.addStep(steps.JSONPropertiesDownload(workerdest="build-properties.json")):bb:step:`JSONPropertiesDownload` transfers a json-encoded string that represents a dictionary where properties maps to a dictionary of build property name to property value; and sourcestamp represents the build's sourcestamp.