WWW

History and Motivation

One of the goals of the 'nine' project is to rework Buildbot's web services to use a more modern, consistent design and implement UI features in client-side JavaScript instead of server-side Python.

The rationale behind this is that a client side UI relieves pressure on the server while being more responsive for the user. The web server only concentrates on serving data via a REST interface wrapping the Data_API. This removes a lot of sources of latency where, in previous versions, long synchronous calculations were made on the server to generate complex pages.

Another big advantage is live updates of status pages, without having to poll or reload. The new system uses Comet techniques in order to relay Data API events to connected clients.

Finally, making web services an integral part of Buildbot, rather than a status plugin, allows tighter integration with the rest of the application.

Design Overview

The www service exposes three pieces via HTTP:

A REST interface wrapping Data_API;
HTTP-based messaging protocols wrapping the Messaging_and_Queues interface; and
Static resources implementing the client-side UI.

The REST interface is a very thin wrapper: URLs are translated directly into Data API paths, and results are returned directly, in JSON format. It is based on JSON API. Control calls are handled with a simplified form of JSONRPC 2.0.

The message interface is also a thin wrapper around Buildbot's MQ mechanism. Clients can subscribe to messages, and receive copies of the messages, in JSON, as they are received by the buildmaster.

The client-side UI is an AngularJS application. Buildbot uses the Python setuptools entry-point mechanism to allow multiple packages to be combined into a single client-side experience. This allows frontend developers and users to build custom components for the web UI without hacking Buildbot itself.

Python development and AngularJS development are very different processes, requiring different environment requirements and skillsets. To maximize hackability, Buildbot separates the two cleanly. An experienced AngularJS hacker should be quite comfortable in the www/ directory, with a few exceptions described below. Similarly, an experienced Python hacker can simply download the pre-built web UI (from pypi!) and never venture near the www/ directory.

URLs

The Buildbot web interface is rooted at its base URL, as configured by the user. It is entirely possible for this base URL to contain path components, e.g., http://build.example.org/buildbot/, if hosted behind an HTTP proxy. To accomplish this, all URLs are generated relative to the base URL.

Overall, the space under the base URL looks like this:

/ -- The HTML document that loads the UI
/api/v{version} -- The root of the REST APIs, each versioned numerically. Users should, in general, use the latest version.
/ws -- The WebSocket endpoint to subscribe to messages from the mq system.
/sse -- The server sent event endpoint where clients can subscribe to messages from the mq system.

REST API

The REST API is a thin wrapper around the data API's "Getter" and "Control" sections. It is also designed, in keeping with REST principles, to be discoverable. As such, the details of the paths and resources are not documented here. Begin at the root URL, and see the Data_API documentation for more information.

Versions

The API described here is version 2. The ad-hoc API from Buildbot-0.8.x, version 1, is no longer supported.

Getting

To get data, issue a GET request to the appropriate path. For example, with a base URL of http://build.example.org/buildbot, the list of masters for builder 9 is available at http://build.example.org/buildbot/api/v2/builder/9/master.

Results are formatted in keeping with the JSON API specification. The top level of every response is an object. Its keys are the plural names of the resource types, and the values are lists of objects, even for a single-resource request. For example:

{
  "meta": {
    "links": [
      {
        "href": "http://build.example.org/api/v2/scheduler",
        "rel": "self"
      }
    ],
    "total": 2
  },
  "schedulers": [
    {
      "link": "http://build.example.org/api/v2/scheduler/1",
      "master": null,
      "name": "smoketest",
      "schedulerid": 1
    },
    {
      "link": "http://build.example.org/api/v2/scheduler/4",
      "master": {
        "active": true,
        "last_active": 1369604067,
        "link": "http://build.example.org/api/v2/master/1",
        "masterid": 1,
        "name": "master3:/BB/master"
      },
      "name": "goaheadtryme",
      "schedulerid": 2
    }
  ]
}

A response may optionally contain extra, related resources beyond those requested. The meta key contains metadata about the response, including navigation links and the total count of resources in a collection.

Several query parameters may be used to affect the results of a request. These parameters are applied in the order described (so, it is not possible to sort on a field that is not selected, for example).

Field Selection

If only certain fields of each resource are required, the field query parameter can be used to select them. For example, the following will select just the names and id's of all schedulers:

http://build.example.org/api/v2/scheduler?field=name&field=schedulerid

Field selection can be used for either detail (single-entity) or collection (multi-entity) requests. The remaining options only apply to collection requests.

Filtering

Collection responses may be filtered on any simple top-level field.

To select records with a specific value use the query parameter {field}={value}. For example, http://build.example.org/api/v2/scheduler?name=smoketest selects the scheduler named "smoketest".

Filters can use any of the operators listed below, with query parameters of the form {field}__{operator}={value}.

eq: equality, or with the same parameter appearing multiple times, set membership
ne: inequality, or set exclusion
lt: select resources where the field's value is less than {value}
le: select resources where the field's value is less than or equal to {value}
gt: select resources where the field's value is greater than {value}
ge: select resources where the field's value is greater than or equal to {value}

For example:

http://build.example.org/api/v2/builder?name__lt=cccc
http://build.example.org/api/v2/buildsets?complete__eq=false

Boolean values can be given as on/off, true/false, yes/no, or 1/0.

Sorting

Collection responses may be ordered with the order query parameter. This parameter takes a field name to sort on, optionally prefixed with - to reverse the sort. The parameter can appear multiple times, and will be sorted lexically with the fields arranged in the given order. For example:

http://build.example.org/api/v2/buildrequest?order=builderid&order=buildrequestid

Pagination

Collection responses may be paginated with the offset and limit query parameters. The offset is the 0-based index of the first result to included, after filtering and sorting. The limit is the maximum number of results to return. Some resource types may impose a maximum on the limit parameter; be sure to check the resulting links to determine whether further data is available. For example:

http://build.example.org/api/v2/buildrequest?order=builderid&limit=10
http://build.example.org/api/v2/buildrequest?order=builderid&offset=20&limit=10

Controlling

Data API control operations are handled by POST requests using a simplified form of JSONRPC 2.0. The JSONRPC "method" is mapped to the data API "action", and the parameters are passed to that application.

The following parts of the protocol are not supported:

positional parameters
batch requests

Requests are sent as an HTTP POST, containing the request JSON in the body. The content-type header must be application/json.

A simple example:

POST http://build.example.org/api/v2/scheduler/4
--> {"jsonrpc": "2.0", "method": "force", "params": {"revision": "abcd", "branch": "dev"}, "id": 843}
<-- {"jsonrpc": "2.0", "result": {"buildsetid": 44}, "id": 843}

Discovery

The Data API provides a discovery endpoint which exposes all endpoints of the API in a JSON format so that one can write middleware to automatically create higher level API, or generate fake data for development. The endpoint is available at:

GET http://build.example.org/api/v2/application.spec

This metadata is guaranteed to be correct, as this is generated from the spec used in data's unit tests. See Adding-Fields-to-Resource-Types for more details on the type system used.

The data validation type system is serialized into JSON in a very simple way. The API returns a list of endpoint specs, each of the form:

{
  path: "<endpoint_path>"
  type: "<endpoint_entity_type>"
  type_spec: "<endpoint_entity_type_spec>"
}

The type spec encoding can have several forms:

Entity or Dict

{
    ..
    type_spec: {
        type: "<type name>"
        fields: [
            {
            name: "<field name>"
            type: "<field type name>"
            type_spec: "<field type spec>"
            }, // [...]
        ]
    }
}

List

{
    ..
    type_spec: {
    type: "list"
    of: {
        type: "<field type name>"
        type_spec: "<field type spec>"
    }
}

Links

{
    ..
    type_spec: {
    type: "link"
    link_specs: [
        "<ep1 path>",
        "<ep2 path>", // [...]
    ]
}

Other base types

{
    ..
    type_spec: {
    type: "(string|integer|boolean|binary|identifier|jsonobject|sourced-properties)"
}

Server-Side Session

The web server keeps a session state for each user, keyed on a session cookie. This session is available from request.getSession(), and data is stored as attributes. The following attributes may be available:

user_info

A dictionary maintained by the authentication subsystem <auth>. It may have the following information about the logged-in user:

username
email
full_name
groups (a list of group names)

As well as additional fields specific to the user info implementation.

The contents of the user_info dictionary are made available to the UI as config.user.

Message API

Currently messages are implemented with two protocols: WebSockets and server sent event. This may be supplemented with other mechanisms before release.

WebSocket

WebSocket is a protocol for arbitrary messaging to and from browser. As an HTTP extension, the protocol is not yet well supported by all HTTP proxy technologies. Although, it has been reported to work well used behind the https protocol. Only one WebSocket connection is needed per browser.

Client can connect using url ws[s]://<BB_BASE_URL>/ws

The protocol used is a simple in-house protocol based on json. Structure of a command from client is as following:

{ "cmd": "<command name>", '_id': <id of the command>, "arg1": arg1, "arg2": arg2 }

cmd is use to reference a command name
_id is used to track the response, can be any unique number or string. Generated by the client. Needs to be unique per websocket session.

Response is sent asynchronously, reusing _id to track which command is responded.

Success answer example would be:

{ "msg": "OK", '_id': 1, code=200 }

Error answer example would be:

{"_id":1,"code":404,"error":"no such command \'poing\'"}

Client can send several command without waiting response.

Responses are not guaranteed to be sent in order.

Several command are implemented:

ping

{"_id":1,"cmd":"ping"}

server will respond with a "pong" message:

{"_id":1,"msg":"pong","code":200}

startConsuming

start consuming events that match path. path are described in the Messaging_and_Queues section. For size optimization reasons, path are encoded joined with "/", and with None wildcard replaced by '*'.

{"_id":1,"cmd":"startConsuming", "path": "change/*/*"}

Success answer example will be:

{ "msg": "OK", '_id': 1, code=200 }

stopConsuming

stop consuming events that was previously registered with path.

{"_id":1,"cmd":"stopConsuming", "path": "change/*/*"}

Success answer example will be:

{ "msg": "OK", '_id': 1, code=200 }

Client will receive events as websocket frames encoded in json with following format:

{"k":key,"m":message}

Server Sent Events

SSE is a simpler protocol than WebSockets and is more REST compliant. It uses the chunk-encoding HTTP feature to stream the events. SSE also does not works well behind enterprise proxy, unless you use the https protocol

Client can connect using following endpoints

http[s]://<BB_BASE_URL>/sse/listen/<path>: Start listening to events on the http connection. Optionally setup a first event filter on <path>. The first message send is a handshake, giving a uuid that can be used to add or remove event filters.
http[s]://<BB_BASE_URL>/sse/add/<uuid>/<path>: Configure a sse session to add an event filter
http[s]://<BB_BASE_URL>/sse/remove/<uuid>/<path>: Configure a sse session to remove an event filter

Note that if a load balancer is setup as a front end to buildbot web masters, the load balancer must be configured to always use the same master given a client ip address for /sse endpoint.

Client will receive events as sse events, encoded with following format:

event: event
data: {'key': <key>, 'message': <message>}

The first event received is a handshake, and is used to inform the client about uuid to use for configuring additional event filters

event: handshake
data: <uuid>

JavaScript Application

The client side of the web UI is written in JavaScript and based on the AngularJS framework and concepts.

This is a Single Page Application" All Buildbot pages are loaded from the same path, at the master's base URL. The actual content of the page is dictated by the fragment in the URL (the portion following the # character). Using the fragment is a common JS techique to avoid reloading the whole page over HTTP when the user changes the URI or clicks a link.

AngularJS

The best place to learn about AngularJS is its own documentation,

AngularJS strong points are:

A very powerful MVC system allowing automatic update of the UI, when data changes
A Testing Framework and philosophy
A deferred system similar to the one from Twisted.
A fast growing community and ecosystem

On top of Angular we use nodeJS tools to ease development

gulp buildsystem, seemlessly build the app, can watch files for modification, rebuild and reload browser in dev mode. In production mode, the buildsystem minifies html, css and js, so that the final app is only 3 files to download (+img).
coffeescript, a very expressive langage, preventing some of the major traps of JS.
jade template langage, adds syntax sugar and readbility to angular html templates.
Bootstrap is a css library providing know good basis for our styles.
Font Awesome is a coherent and large icon library

modules we may or may not want to include:

momentjs is a library implementing human readable relative timings (e.g. "one hour ago")
ngGrid is a grid system for full featured searcheable/sortable/csv exportable grids
angular-UI is a collection of jquery based directives and filters. Probably not very useful for us
JQuery the well known JS framework, allows all sort of dom manipulation. Having it inside allows for all kind of hacks we may want to avoid.

Extensibility

Buildbot UI is designed for extensibility. The base application should be pretty minimal, and only include very basic status pages. Base application cannot be disabled so any page not absolutely necessary should be put in plugins.

Some Web plugins are maintained inside buildbot's git repository, but this is absolutely not necessary. Unofficial plugins are encouraged, please be creative!

Please look at official plugins for working samples.

Typical plugin source code layout is:

setup.py                     # standard setup script. Most plugins should use the same boilerplate, which helps building guanlecoja app as part of the setup. Minimal adaptation is needed
<pluginname>/__init__.py     # python entrypoint. Must contain an "ep" variable of type buildbot.www.plugin.Application. Minimal adaptation is needed
guanlecoja/config.coffee     # Configuration for guanlecoja. Few changes are needed here. Please see guanlecoja docs for details.
src/..                       # source code for the angularjs application. See guanlecoja doc for more info of how it is working.
package.json                 # declares npm dependency. normallly, only guanlecoja is needed. Typically, no change needed
gulpfile.js                  # entrypoint for gulp, should be a one line call to guanlecoja. Typically, no change needed
MANIFEST.in                  # needed by setup.py for sdist generation. You need to adapt this file to match the name of your plugin

Plugins are packaged as python entry-points for the buildbot.www namespace. The python part is defined in the buildbot.www.plugin module. The entrypoint must contain a twisted.web Resource, that is populated in the web server in /<pluginname>/.

The front-end part of the plugin system automatically loads /<pluginname>/scripts.js and /<pluginname>/styles.css into the angular.js application. The scripts.js files can register itself as a dependency to the main "app" module, register some new states to $stateProvider, or new menu items via glMenuProvider.

The entrypoint containing a Resource, nothing forbids plugin writers to add more REST apis in /<pluginname>/api. For that, a reference to the master singleton is provided in master attribute of the Application entrypoint. You are even not restricted to twisted, and could even load a wsgi application using flask, django, etc.

Routing

AngularJS uses router to match URL and choose which page to display. The router we use is ui.router. Menu is managed by guanlecoja-ui's glMenuProvider. Please look at ui.router, and guanlecoja-ui documentation for details.

Typically, a route regitration will look like following example.

# ng-classify declaration. Declares a config class
class State extends Config
    # Dependancy injection: we inject $stateProvider and glMenuServiceProvider
    constructor: ($stateProvider, glMenuServiceProvider) ->

        # Name of the state
        name = 'console'

        # Menu configuration.
        glMenuServiceProvider.addGroup
            name: name
            caption: 'Console View'     # text of the menu
            icon: 'exclamation-circle'  # icon, from Font-Awesome
            order: 5                    # order in the menu, as menu are declared in several places, we need this to control menu order

        # Configuration for the menu-item, here we only have one menu item per menu, glMenuProvider won't create submenus
        cfg =
            group: name
            caption: 'Console View'

        # Register new state
        state =
            controller: "#{name}Controller"
            controllerAs: "c"
            templateUrl: "console_view/views/#{name}.html"
            name: name
            url: "/#{name}"
            data: cfg

        $stateProvider.state(state)

Directives

We use angular directives as much as possible to implement reusable UI components.

Services

BuildbotService

BuildbotService is the base service for accessing to the Buildbot data API. It uses and is derivated from restangular. Restangular offers nice semantics around nested REST endpoints. Please see restangular documentation for overview on how it works.

BuildbotService adds serveral methods to restangular objects in order to integrate it with EventSource. The idea is to simplifify automatic update of the $scope based on events happening on a given data endpoint

# Following code will get initial data from 'api/v2/build/1/step/2'
# and register to events from 'sse/build/1/step/2'
# Up to the template to specify what to display

buildbotService.one("build", 1).one("step", 2).bind($scope)

Difference with restangular is all restangular objects are reused, i.e. if you are calling bind() twice on the same object, no additionnal ressource is gathered via http.

Several methods are added to each "restangularized" objects, aside from get(), put(), delete(), etc.:

.bind($scope, opts)

Bind the api results to the $scope, automatically listening to events on this endpoint, and modifying the $scope object accordingly. This method automatically references the scopes where the data is used, and will remove the reference when the $scope is destoyed. When no scope is referencing the data anymore, the service will wait a configurable amount of time, and stop listening to associated events. As a result, the service will loose real-time track of the underlying data, so any subsequent call to bind() will trigger another http requests to get updated data. This delayed event unregister mechanism enables better user experience. When user is going back and forth between several pages, chances are that the data is still on-track, so the page will be displayed instantly.

bind() takes several optional parameters in opts:

dest (default: $scope)

object where to store the results

ismutable: (elem) -> boolean (default: always false)

function used to know if the object will not evolve anymore (so no need to register to events)

onchild: (child) ->

function called for each child, at init time, and when new child is detected through events. This can be used to get more data derived from a list. The child received are restangular elements

.on(eventtype, callback)

Listen to events for this endpoint. When bind() semantic is not useful enough, you can use this lower level api.

.some(route, queryParams)

like .all(), but allows to specify query parameters

queryParams

query parameters used to filter the results of a list api

.control(method, params)

Call the control data api. This builds up a POST with jsonapi encoded parameters

DataService

DataService is the future replacement of BuildbotService for accessing the Buildbot data API. It has a modern interface for accessing data. It uses IndexedDB for storing cached data as a single data store, and LocalStorage for broadcasting events between browser tabs. DataService works in a master/slave architecture. The master browser tab is responsible for keeping the requested data up to date in the IndexedDB and notify slaves when a data is ready to be used or it is updated. It handles both the Rest API calls and the WebSocket subscriptions globally.

It uses the following libraries:

Dexie.js (https://github.com/dfahlander/Dexie.js) - Minimalistic IndexedDB API with bulletproof transactions
Tabex (https://github.com/nodeca/tabex) - Master election and in browser message bus

The DataService is available as a standalone AngularJS module. Installation via bower:

bower install buildbot-data --save

Inject the bbData module to your application:

angular.module('myApp', ['bbData'])

Methods:

.getXs([id], [query]): returns a promise<Collection>, when the promise is resolved, the Collection contains all the requested data

it's highly advised to use these instead of the lower level .get('string') function

Xs can be the following: Builds, Builders, Buildrequests, Buildsets, Buildslaves, Changes, Changesources, Forceschedulers, Masters, Schedulers, Sourcestamps

call .getArray() on the returned promise to get the Collection before it's filled with the initial data
# assign builds to $scope.builds once the Collection is filled
dataService.getBuilds(builderid: 1).then (builds) ->
    $scope.builds = builds
    # load steps for every build
    builds.forEach (b) -> b.loadSteps()

# assign builds to $scope.builds before the Collection is filled using the .getArray() function
$scope.builds = dataService.getBuilds(builderid: 1).getArray()

.get(endpoint, [id], [query]): returns a promise<Collection>, when the promise is resolved, the Collection contains all the requested data

call .getArray() on the returned promise to get the Collection before it's filled with the initial data

# assign builds to $scope.builds once the Collection is filled
builderid = 1
dataService.get("builders/#{builderid}/builds", limit: 1).then (builds) ->
    $scope.builds = builds
    # load steps for every build
    builds.forEach (b) -> b.loadSteps()

# assign builds to $scope.builds before the Collection is filled using the .getArray() function
$scope.builds = dataService.get('builds', builderid: 1).getArray()

.open(scope): returns a DataAccessor, handles bindings

open a new accessor every time you need updating data in a controller
it registers a $destroy event handling function on the scope, it automatically unsubscribes from updates that has been requested by the DataAccessor

# open a new accessor every time you need updating data in a controller
class DemoController extends Controller
    constructor: ($scope, dataService) ->
        # automatically closes all the bindings when the $scope is destroyed
        opened = dataService.open($scope)
        # alternative syntax:
        #   opened = dataService.open()
        #   opened.closeOnDestroy($scope)
        # closing it manually is also possible:
        #   opened.close()

        # request new data, it updates automatically
        @builders = opened.getBuilders(limit: 10, order: '-started_at').getArray()

.control(url, method, [params]): returns a promise, sends a JSON RPC2 POST request to the server

# open a new accessor every time you need updating data in a controller
dataService.control('forceschedulers/force', 'force').then (response) ->
    $log.debug(response)
, (reason) ->
    $log.error(reason)

.clearCache(): clears the IndexedDB tables and reloads the current page

class DemoController extends Controller
    constructor: (@dataService) ->
    onClick: -> @dataService.clearCache()

Methods on the object returned by the .getXs() methods:

.getXs([id], [query]): returns a promise<Collection>, when the promise is resolved, the Collection contains all the requested data

same as dataService.getXs, but with relative endpoint

# assign builds to $scope.builds once the Collection is filled
dataService.getBuilds(builderid: 1).then (builds) ->
    $scope.builds = builds
    # get steps for every build
    builds.forEach (b) ->
        b.getSteps().then (steps) ->
            # assign completed test to every build
            b.complete_steps = steps.map (s) -> s.complete

.loadXs([id], [query]): returns a promise<Collection>, the Collection contains all the requested data, which is also assigned to o.Xs

o.loadXs() is equivalent to o.getXs().then (xs) -> o.xs = xs

$q (resolve) ->
    # get builder with id = 1
    dataService.getBuilders(1).then (builders) ->
        builders.forEach (builder) ->
            # load all builds
            builder.loadBuilds().then (builds) ->
                builds.forEach (build) ->
                    # load all buildsteps
                    build.loadSteps().then -> resolve(builders[0])
.then (builder) ->
    # builder has a builds field, and the builds have a steps field containing the corresponding data
    $log.debug(builder)

.control(method, params): returns a promise, sends a JSON RPC2 POST request to the server

RecentStorage

The service provides methods for adding, retrieving and clearing recently viewed builders and builds. It uses IndexedDB to store data inside the user’s browser. You can see the list of supported browsers here: http://caniuse.com/indexeddb.

builder and build object properties:

link - string: this specifies the builder’s or build’s link
caption - string: this specifies the builder’s or build’s shown caption

Sample:

{
    link: '#/builders/1'
    caption: 'Mac'
}

Methods:

.addBuild(build): stores the build passed as argument
.addBuild(builder): stores the builder passed as argument
.getBuilds(): returns a promise, the result will be an array of builds when the promise is resolved example:
```
recentStorage.getBuilds().then (e) ->
    $scope.builds = e
```
.getBuilders(): returns a promise, the result will be an array of builders when the promise is resolved example:
```
recentStorage.getBuilders().then (e) ->
    $scope.builders = e
```

.getAll(): returns a promise, the result will be an object with two fields, recent_builds and recent_builders example:

recentStorage.getAll().then (e) ->
    $scope.builds = e.recent_builds
    $scope.builders = e.recent_builders

.clearAll(): removes the stored builds and builders example:
```
recentStorage.clearAll()
```

Mocks and testing utils

httpMock.coffee

This modules adds decorateHttpBackend($httpBackend) to the global namespace. This function decorate the $httpBackend with additional functionality:

.expectDataGET(ep, {nItems:<int or undefined>, override: <fn or undefined>})
Automatically create a GET expectation to the data api, given the data spec Available options are:
- nItems: if defined, this will generate a collection of nItems instead of single value
- override: a custom function to override the resulting generated data
Example: $httpBackend.expectDataGET("change", {nItems:2, override: (val) -> val[1].id=4 }) will create 2 changes, but the id of the second change will be overridden to 4

Linking with Buildbot

A running buildmaster needs to be able to find the JavaScript source code it needs to serve the UI. This needs to work in a variety of contexts - Python development, JavaScript development, and end-user installations. To accomplish this, the gulp build process finishes by bundling all of the static data into a Python distribution tarball, along with a little bit of Python glue. The Python glue implements the interface described below, with some care taken to handle multiple contexts.

Hacking Quick-Start

This section describes how to get set up quickly to hack on the JavaScript UI. It does not assume familiarity with Python, although a Python installation is required, as well as virtualenv. You will also need NodeJS, and npm installed.

Prerequisites

Install latest release of node.js.

http://nodejs.org/ is a good start for windows and osx.

For linux, as node.js is evolving very fast, distros versions are often too old. For ubuntu, for example, you want to use following ppa:
```
sudo add-apt-repository -y ppa:chris-lea/node.js
```
Please feel free to update this documentation for other distros.
Install gulp globally. Gulp is the build system used for coffeescript development.
```
sudo npm install -g gulp
```

Hacking the Buildbot JavaScript

To effectively hack on the Buildbot JavaScript, you'll need a running Buildmaster, configured to operate out of the source directory (unless you like editing minified JS). Start by cloning the project and its git submodules:

git clone git://github.com/buildbot/buildbot.git

In the root of the source tree, create and activate a virtualenv to install everything in:

virtualenv sandbox
source sandbox/bin/activate

This creates an isolated Python environment in which you can install packages without affecting other parts of the system. You should see (sandbox) in your shell prompt, indicating the sandbox is activated.

Next, install the Buildbot-WWW and Buildbot packages using --editable, which means that they should execute from the source directory.

pip install --editable pkg
pip install --editable master/
make frontend

This will fetch a number of dependencies from pypi, the Python package repository. This will also fetch a bunch a bunch of node.js dependencies used for building the web application, and a bunch of client side js dependencies, with bower

Now you'll need to create a master instance. For a bit more detail, see the Buildbot tutorial (first-run-label).

buildbot create-master sandbox/testmaster
mv sandbox/testmaster/master.cfg.sample sandbox/testmaster/master.cfg
buildbot start sandbox/testmaster

If all goes well, the master will start up and begin running in the background. As you just installed www in editable mode (aka 'develop' mode), setup.py did build the web site in prod mode, so the everything is minified, making it hard to debug.

When doing web development, you usually run:

cd www/base
gulp dev

This will compile the base webapp in development mode, and automatically rebuild when files change.

Testing with real data

Front-end only hackers might want to just skip the master and slave setup, and just focus on the UI. It can also be very useful to just try the UI with real data from your production. For those use-cases, gulp dev proxy can be used.

This tool is a small nodejs app integrated in the gulp build that can proxy the data and websocket api from a production server to your development environment. Having a proxy is slightly slower, but this can be very useful for testing with real complex data.

You still need to have python virtualenv configured with master package installed, like we described in previous paragraph.

Provided you run it in a buildbot master virtualenv, the following command will start the UI and redirect the api calls to the nine demo server:

gulp dev proxy --host nine.buildbot.net

You can then just point your browser to localhost:8010, and you will access nine.buildbot.net, with your own version of the UI.

Guanlecoja

Buildbot's build environment has been factorized for reuse in other projects and plugins, and is callsed Guanlecoja.

The documentation and meaning of this name is maintained in Guanlecoja's own site. https://github.com/buildbot/guanlecoja/

Testing Setup

buildbot_www uses Karma to run the coffeescript test suite. This is the official test framework made for angular.js. We don't run the front-end testsuite inside the python 'trial' test suite, because testing python and JS is technically very different.

Karma needs a browser to run the unit test in. It supports all the major browsers. Given our current experience, we did not see any bugs yet that would only happen on a particular browser this is the reason that at the moment, only headless browser "PhantomJS" is used for testing.

We enforce that the tests are run all the time after build. This does not impact the build time by a great factor, and simplify the workflow.

In some case, this might not be desirable, for example if you run the build on headless system, without X. PhantomJS, even if it is headless needs a X server like xvfb. In the case where you are having difficulties to run Phantomjs, you can build without the tests using the command:

gulp prod --notests

Debug with karma

console.log is available via karma. In order to debug the unit tests, you can also use the global variable dump, which dumps any object for inspection in the console. This can be handy to be sure that you dont let debug logs in your code to always use dump

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

www.rst

www.rst

WWW

History and Motivation

Design Overview

URLs

REST API

Versions

Getting

Field Selection

Filtering

Sorting

Controlling

Discovery

Server-Side Session

WebSocket

Server Sent Events

JavaScript Application

AngularJS

Extensibility

Routing

Directives

Services

BuildbotService

DataService

RecentStorage

Mocks and testing utils

httpMock.coffee

Linking with Buildbot

Hacking Quick-Start

Prerequisites

Hacking the Buildbot JavaScript

Testing with real data

Guanlecoja

Testing Setup

Debug with karma

Files

www.rst

Latest commit

History

www.rst

File metadata and controls

WWW

History and Motivation

Design Overview

URLs

REST API

Versions

Getting

Field Selection

Filtering

Sorting

Pagination

Controlling

Discovery

Server-Side Session

WebSocket

Server Sent Events

JavaScript Application

AngularJS

Extensibility

Routing

Directives

Services

BuildbotService

DataService

RecentStorage

Mocks and testing utils

httpMock.coffee

Linking with Buildbot

Hacking Quick-Start

Prerequisites

Hacking the Buildbot JavaScript

Testing with real data

Guanlecoja

Testing Setup

Debug with karma