developer.texinfo

This chapter is the official repository for the collected wisdom of the
Buildbot hackers.

It contains some sparse documentation of the inner workings of Buildbot, but of
course, the final reference for that is the source itself.

More importantly, this chapter represents the official repository of all
agreed-on patterns for use in Buildbot.  In this case, the source is a
@i{terrible} reference, because much of it is old and crusty.  But we are
trying to do things the new, better way, and those new, better ways are
described here.

@menu
* Buildmaster Service Hierarchy::
* Utilities::
* The Database::
* The Event Loop::
* String Types::
* Subscription Interfaces::
* Jinja Web Templates::
* Twisted Idioms::
* Testing Idioms and Utilities::
* Developer Class Index::
@end menu

@node Buildmaster Service Hierarchy
@section Buildmaster Service Hierarchy

TODO

@node Utilities
@section Utilities

@menu
* buildbot.util.collections::
* buildbot.util.eventual::
@end menu

Several small utilities are available at the top-level @code{buildbot.util}
package.  As always, see the API documentation for more information.

@table @code

@item natualSort

This function sorts strings "naturally", with embedded numbers sorted
numerically.  This ordering is good for objects which might have a numeric
suffix, e.g., @code{winslave1}, @code{winslave2}, ..

@item formatInterval

This function will return a human-readable string describing a length of time,
given a number of seconds.

@item ComparableMixin

This mixin class adds comparability to a subclass.  Use it like this:

@example
class Widget(FactoryProduct, ComparableMixin):
    compare_attrs = [ 'radius', 'thickness' ]
    # ...
@end example

Any attributes not in @code{compare_attrs} will not be considered when
comparing objects.  This is particularly useful in implementing buildbot's
reconfig logic, where a simple comparison between the new and existing objects
can determine whether the new object should replace the existing object.

@item safeTranslate

This function will filter out some inappropriate characters for filenames; it
is suitable for adapting strings from the configuration for use as filenames.
It is not suitable for use with strings from untrusted sources.

@item LRUCache

This is a simple least-recently-used cache.  Its constructor takes a maximum
size.  When the cache grows beyond this size, the least-recently used items
will be automatically removed from the cache.  The class has @code{get} and
@code{add} methods, and can also be accessed via dictionary syntax
(@code{lru['id']}).

@end table

@node buildbot.util.collections
@subsection buildbot.util.collections

This package provides a few useful collection objects.

For compatibility, it provides a clone of the Python
@code{collections.defaultdict} for use in Python-2.4.  In later versions, this
is simply a reference to the built-in @code{defaultdict}, so buildbot code can
simply use @code{buildbot.util.collections.defaultdict} everywhere.

It also provides a @code{KeyedSets} class that can represent any numbers of
sets, keyed by name (or anything hashable, really).  The object is specially
tuned to contain many different keys over its lifetime without wasting memory.
See the docstring for more information.

@node buildbot.util.eventual
@subsection buildbot.util.eventual

This package provides a simple way to say "please do this later":

@example
from buildbot.util.eventual import eventually
def do_what_I_say(what, where):
    # ...
eventually(do_what_I_say, "clean up", "your bedroom")
@end example

The package defines "later" as "next time the reactor has control", so this is
a good way to avoid long loops that block other activity in the reactor.
Callables given to @code{eventually} are guaranteed to be called in the same
order as the calls to @code{eventually}.  Any errors from the callable are
logged, but will not affect other callables.

If you need a deferred that will fire "later", use @code{fireEventually}.  This
function returns a deferred that will not errback.

@node The Database
@section The Database

@dvindex buildbot.db.dbspec.DBSpec
@dvindex buildbot.db.connector.DBConnector

Buildbot stores its state in a database, using the Python DBAPI to access it.

A database is specified with the @code{buildbot.db.dbspec.DBSpec} class, which
encapsulates all of the parameters necessary to create a DBAPI connection.  The
DBSpec class can create either a single synchronous connection, or a twisted
adbapi connection pool.

Most uses of the database in Buildbot are done through the
@code{buildbot.db.connector.DBConnector} class, which wraps the DBAPI to
provide purpose-specific functions.

The database schema is managed by a special class, described in the next
section.

@menu
* Database Schema::
* Changing the Schema::
@end menu

@node Database Schema
@subsection Database Schema

@dvindex buildbot.db.schema.DBSchemaManager

The SQL for the database schema is available in
@code{buildbot/db/schema/tables.sql}.  However, note that this file is not used
for new installations or upgrades of the Buildbot database.

Instead, the @code{buildbot.db.schema.DBSchemaManager} handles this task.  The
operation of this class centers around a linear sequence of database versions.
Versions start at 0, which is the old pickle-file format.  The manager has
methods to query the version of the database, and the current version from the
source code.  It also has an @code{upgrade} method which will upgrade the
database to the latest version.  This operation is currently irreversible.

There is no operation to "install" the latest schema.  Instead, a fresh install
of buildbot begins with an (empty) version-0 database, and upgrades to the
current version.  This trades a bit of efficiency at install time for
assurances that the upgrade code is well-tested.

@node Changing the Schema
@subsection Changing the Schema

To make a change to the database schema, follow these steps:

@enumerate

@item

Increment @code{CURRENT_VERSION} in @code{buildbot/db/schema/manager.py} by
one.  This is your new version number.

@item

Create @file{buildbot/db/schema/vN.py}, where N is your version number, by
copying the previous script and stripping it down.  This script should define a
subclass of @code{buildbot.db.schema.base.Updater} named @code{Updater}. 

The class must define the method @code{upgrade}, which takes no arguments.  It
should upgrade the database from the previous version to your version,
including incrementing the number in the @code{VERSION} table, probably with an
@code{UPDATE} query.

Consult the API documentation for the base class for information on the
attributes that are available.

@item

Edit @file{buildbot/test/unit/test_db_schema_master.py}.  If your upgrade
involves moving data from the basedir into the database proper, then edit
@code{fill_basedir} to add some test data.

Add code to @code{assertDatabaseOKEmpty} to check that your upgrade works on an
empty database.

Add code to @code{assertDatabaseOKFull} to check that your upgrade works on a
database with pre-existing data.  Do this even if your changes do not move any
data from the basedir.

Run the tests to find the bugs you introduced in step 2.

@item

Increment the version number in the @code{test_get_current_version} test in the
same file.  Only do this after you've finished the previous step - a failure of
this test is a good reminder that testing isn't done yet.

@item

Finally, make the corresponding changes to @file{buildbot/db/schema/tables.sql}.

@end enumerate

@node The Event Loop
@section The Event Loop

TODO

@node String Types
@section String Types

TODO

@node Subscription Interfaces
@section Subscription Interfaces

TODO use @code{buildbot.eventually}

@node Jinja Web Templates
@section Jinja Web Templates

Buildbot uses Jinja2 to render its web interface.  The authoritative source for
this templating engine is @uref{http://jinja.pocoo.org/2/documentation/, its
own documentation}, of course, but a few notes are in order for those who are
making only minor modifications.

@heading Whitespace

Jinja directives are enclosed in @code{@{% .. %@}}, and sometimes also have
dashes.  These dashes strip whitespace in the output.  For example:

@example
@{% for entry in entries %@}
  <li>@{@{ entry @}@}</li>
@{% endfor %@}
@end example

will produce output with too much whitespace:

@example

  <li>pigs</li>


  <li>cows</li>

@end example

But adding the dashes will collapse that whitespace completely:

@example
@{% for entry in entries -%@}
  <li>@{@{ entry @}@}</li>
@{%- endfor %@}
@end example

yields

@example
<li>pigs</li><li>cows</li>
@end example

@node Twisted Idioms
@section Twisted Idioms

TODO

@node Testing Idioms and Utilities
@section Testing Idioms and Utilities

@menu
* Keeping State in Tests::
@end menu

@node Keeping State in Tests
@subsection Keeping State in Tests

Python does not allow assignment to anything but the innermost local scope or
the global scope with the @code{global} keyword.  This presents a problem when
creating nested functions:

@example
def test_localVariable(self):
    cb_called = False
    def cb():
        cb_called = True
    cb()
    self.assertTrue(cb_called) # will fail!
@end example

The @code{cb_called = True} assigns to a @i{different variable} than
@code{cb_called = False}.  In production code, it's usually best to work around
such problems, but in tests this is often the clearest way to express the
behavior under test.

The solution is to change something in a common mutable object.  While a simple
list can serve as such a mutable object, this leads to code that is hard to
read.  Instead, use @code{State}:

@example
from buildbot.test.state import State

def test_localVariable(self):
    state = State(cb_called=False)
    def cb():
        state.cb_called = True
    cb()
    self.assertTrue(state.cb_called) # passes
@end example

This is almost as readable as the first example, but it actually works. 

@node Developer Class Index
@section Developer Class Index
@printindex dv