Fetching contributors…
Cannot retrieve contributors at this time
564 lines (489 sloc) 19.8 KB
<?xml version="1.0" encoding="utf-8" ?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "">
<html xmlns="" xml:lang="en" lang="en">
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
<meta name="generator" content="Docutils 0.4:" />
<style type="text/css">
:Author: David Goodger
:Date: $Date: 2005-12-18 01:56:14 +0100 (Sun, 18 Dec 2005) $
:Revision: $Revision: 4224 $
:Copyright: This stylesheet has been placed in the public domain.
Default cascading style sheet for the HTML output of Docutils.
See for how to
customize this style sheet.
/* used to remove borders from tables and images */
.borderless, table.borderless td, table.borderless th {
border: 0 }
table.borderless td, table.borderless th {
/* Override padding for "table.docutils td" with "! important".
The right padding separates the table cells. */
padding: 0 0.5em 0 0 ! important }
.first {
/* Override more specific margin styles with "! important". */
margin-top: 0 ! important }
.last, .with-subtitle {
margin-bottom: 0 ! important }
.hidden {
display: none }
a.toc-backref {
text-decoration: none ;
color: black }
blockquote.epigraph {
margin: 2em 5em ; }
dl.docutils dd {
margin-bottom: 0.5em }
/* Uncomment (and remove this text!) to get bold-faced definition list terms
dl.docutils dt {
font-weight: bold }
div.abstract {
margin: 2em 5em }
div.abstract p.topic-title {
font-weight: bold ;
text-align: center }
div.admonition, div.attention, div.caution, div.danger, div.error,
div.hint, div.important, div.note, div.tip, div.warning {
margin: 2em ;
border: medium outset ;
padding: 1em }
div.admonition p.admonition-title, div.hint p.admonition-title,
div.important p.admonition-title, div.note p.admonition-title,
div.tip p.admonition-title {
font-weight: bold ;
font-family: sans-serif }
div.attention p.admonition-title, div.caution p.admonition-title,
div.danger p.admonition-title, div.error p.admonition-title,
div.warning p.admonition-title {
color: red ;
font-weight: bold ;
font-family: sans-serif }
/* Uncomment (and remove this text!) to get reduced vertical space in
compound paragraphs.
div.compound .compound-first, div.compound .compound-middle {
margin-bottom: 0.5em }
div.compound .compound-last, div.compound .compound-middle {
margin-top: 0.5em }
div.dedication {
margin: 2em 5em ;
text-align: center ;
font-style: italic }
div.dedication p.topic-title {
font-weight: bold ;
font-style: normal }
div.figure {
margin-left: 2em ;
margin-right: 2em }
div.footer, div.header {
clear: both;
font-size: smaller }
div.line-block {
display: block ;
margin-top: 1em ;
margin-bottom: 1em }
div.line-block div.line-block {
margin-top: 0 ;
margin-bottom: 0 ;
margin-left: 1.5em }
div.sidebar {
margin-left: 1em ;
border: medium outset ;
padding: 1em ;
background-color: #ffffee ;
width: 40% ;
float: right ;
clear: right }
div.sidebar p.rubric {
font-family: sans-serif ;
font-size: medium }
div.system-messages {
margin: 5em }
div.system-messages h1 {
color: red }
div.system-message {
border: medium outset ;
padding: 1em }
div.system-message p.system-message-title {
color: red ;
font-weight: bold }
div.topic {
margin: 2em }
h1.section-subtitle, h2.section-subtitle, h3.section-subtitle,
h4.section-subtitle, h5.section-subtitle, h6.section-subtitle {
margin-top: 0.4em }
h1.title {
text-align: center }
h2.subtitle {
text-align: center }
hr.docutils {
width: 75% }
img.align-left {
clear: left }
img.align-right {
clear: right }
ol.simple, ul.simple {
margin-bottom: 1em }
ol.arabic {
list-style: decimal }
ol.loweralpha {
list-style: lower-alpha }
ol.upperalpha {
list-style: upper-alpha }
ol.lowerroman {
list-style: lower-roman }
ol.upperroman {
list-style: upper-roman }
p.attribution {
text-align: right ;
margin-left: 50% }
p.caption {
font-style: italic }
p.credits {
font-style: italic ;
font-size: smaller }
p.label {
white-space: nowrap }
p.rubric {
font-weight: bold ;
font-size: larger ;
color: maroon ;
text-align: center }
p.sidebar-title {
font-family: sans-serif ;
font-weight: bold ;
font-size: larger }
p.sidebar-subtitle {
font-family: sans-serif ;
font-weight: bold }
p.topic-title {
font-weight: bold }
pre.address {
margin-bottom: 0 ;
margin-top: 0 ;
font-family: serif ;
font-size: 100% }
pre.literal-block, pre.doctest-block {
margin-left: 2em ;
margin-right: 2em ;
background-color: #eeeeee }
span.classifier {
font-family: sans-serif ;
font-style: oblique }
span.classifier-delimiter {
font-family: sans-serif ;
font-weight: bold }
span.interpreted {
font-family: sans-serif }
span.option {
white-space: nowrap }
span.pre {
white-space: pre }
span.problematic {
color: red }
span.section-subtitle {
/* font-size relative to parent (h1..h6 element) */
font-size: 80% }
table.citation {
border-left: solid 1px gray;
margin-left: 1px }
table.docinfo {
margin: 2em 4em }
table.docutils {
margin-top: 0.5em ;
margin-bottom: 0.5em }
table.footnote {
border-left: solid 1px black;
margin-left: 1px }
table.docutils td, table.docutils th,
table.docinfo td, table.docinfo th {
padding-left: 0.5em ;
padding-right: 0.5em ;
vertical-align: top }
table.docutils th.field-name, table.docinfo th.docinfo-name {
font-weight: bold ;
text-align: left ;
white-space: nowrap ;
padding-left: 0 }
h1 tt.docutils, h2 tt.docutils, h3 tt.docutils,
h4 tt.docutils, h5 tt.docutils, h6 tt.docutils {
font-size: 100% }
tt.docutils {
background-color: #eeeeee } {
list-style-type: none }
<div class="document" id="pony-build">
<h1 class="title">pony-build</h1>
<p>pony-build is a simple continuous integration package that lets you
run a server to display client build results. It consists of two
components, a server (which is run in some central &amp; accessible
location), and one or more clients (which must be able to contact the
server via HTTP).</p>
<p>Philosophy statement: good development tools for Python should be easy
to install, easy to hack, and not overly constraining. Two out of
three ain't bad ;).</p>
<p>The pony-build architectural model is this: use decoupled components
that communicate via webhooks whenever possible.</p>
<p>Also see <a class="reference" href="">buildbot</a>.</p>
<div class="section">
<h1><a id="links" name="links">Links</a></h1>
<p>pony-build is hosted on github.</p>
<p>pony-build central repository: <a class="reference" href=""></a></p>
<p>pony-build issue tracking: <a class="reference" href=""></a></p>
<p>pony-build wiki: <a class="reference" href=""></a></p>
<p>pony-build mailing list: <a class="reference" href=""></a></p>
<div class="section">
<h1><a id="requirements" name="requirements">Requirements</a></h1>
<dl class="docutils">
<dt>Server side:</dt>
<dd><p class="first">Requires Python 2.6 or above.</p>
<p>Jinja2 (easy_installable).</p>
<p class="last">For the Quixote Web UI, Quixote 2.6 (also easy_installable).</p>
<dt>Client side:</dt>
<dd>Python. Should work down to Python 2.4. Developed with 2.5.</dd>
<div class="section">
<h1><a id="pony-build-server" name="pony-build-server">pony-build server</a></h1>
<p>The command:</p>
<pre class="literal-block">
python -m -f &lt;shelve filename&gt; -p &lt;port&gt;
<p>will run the Quixote-based pony-build Web app on the given port,
reading &amp; writing from the sqlite database in 'filename'. If you omit
the '-f' argument, it will use an in-memory database.</p>
<p>For example,</p>
<pre class="literal-block">
python -m -f test.db -p 8080
<p>will run a server that can be accessed on <a class="reference" href="http://localhost:8080/">http://localhost:8080/</a>. This
server will report on whatever results are sent to it by the client (see
below), based on the package name ('name', below). All data will be
saved to 'test.db'.</p>
<p>See 'architecture, and extending pony-build', below.</p>
<div class="section">
<h1><a id="pony-build-client-scripts" name="pony-build-client-scripts">pony-build client scripts</a></h1>
<div class="section">
<h2><a id="build-scripts" name="build-scripts">Build scripts</a></h2>
<p>Client build scripts are just scripts that set up &amp; run a list of commands:</p>
<pre class="literal-block">
from pony_client import BuildCommand, TestCommand, do, send
name = 'example'
server_url = 'http://localhost:8080/xmlrpc'
commands = [ BuildCommand(['/bin/echo', 'hello, world'], name='step 1'),
TestCommand(['/bin/echo', 'this is a test'], name='step 2')
results = do('package', commands)
send('http://localhost:8080/xmlrpc', results)
<p>Client results are communicated to the server by XML-RPC, so the client
must be able to reach the server via the HTTP protocol.</p>
<p>See <tt class="docutils literal"><span class="pre">client/build-cpython</span></tt> for an example of building a Subversion-based
C project (checkout, configure, make, run tests).</p>
<p>See <tt class="docutils literal"><span class="pre">client/build-pony-build</span></tt> for an example of building a Git-based
Python project (clone, build, run tests).</p>
<p>Note that 'pony_client' doesn't depend on the rest of pony-build, so you
can distribute it with other packages if you want.</p>
<div class="section">
<h2><a id="client-query-scripts" name="client-query-scripts">Client query scripts</a></h2>
<p>Client query scripts request information from the server. For example,
see 'bin/notify-failure-email', which checks the status of the last build
of a particular package and sends an e-mail.</p>
<div class="section">
<h1><a id="architecture-and-extending-pony-build" name="architecture-and-extending-pony-build">Architecture, and extending pony-build</a></h1>
<p>The pony-build server is basically a storage receptacle for &quot;bags&quot; of
key-value pairs. It's easiest point of extension is in the Web
interface, where you can substitute any WSGI app object to serve Web
pages; see 'bin/run-server', and the function call to
'server.create(...)' (aka pony_build.server.create(...)'.</p>
<p>To write a new Web UI, you will need access to the stored pony-build
server data. You should work through the 'PonyBuildCoordinator'
interface in 'pony_build.coordinator'; you can get a handle to the
current coordinator object with 'pony_build.server.get_coordinator()'.
<strong>The coordinator API is will be stable and public</strong>, after suitable
<div class="section">
<h2><a id="client-to-server-communication" name="client-to-server-communication">Client-to-server communication</a></h2>
<p>Each client sends two bags of information: the first, 'client_info',
contains information global to the build client, such as package name,
host name, architecture, and a list of tags. The second,
'results_list', is an ordered list of dictionaries, each one
representing a build step. (The default contents of these dictionaries
are pretty obvious: status, stderr, stdout, etc.) Upon receipt of
this info, the pony-build server creates a third object, a dictionary,
containing information such as the server time at which the result
was received,</p>
<p>These three bags of info -- 'receipt', 'client_info', and
'results_list' -- are it. The coordinator functions give you ways to
slice and dice which results set you want (e.g. latest for a
particular package), and then usually return one or more triples of
(receipt, client_info, results_list).</p>
<p>Clients can send arbitrary key/value pairs in their &quot;bags&quot;; two
simple ways to extend things are to add new k/v pairs for specific
purposes, and/or to use the 'tags' key in the client_info dict.
The 'tags' associated value is a list of strings.</p>
<p>receipt['result_key'] is the internal key used to store the result.</p>
<div class="section">
<h2><a id="notifications" name="notifications">Notifications</a></h2>
<p>RSS2 and pubsubhubbub (<a class="reference" href=""></a>) are the
core of the notification system built into pony_build; the &quot;officially
correct&quot; way to actively notify interested people of build results is
to publish them via RSS2, push them to a pubsubhubbub server, and let
someone else deal with translating those into e-mail alerts, etc.</p>
<p>All of the RSS feeds that pony-build makes available can be posted to
pubsubhubbub with the proper configuration (see -P and -S options to
<tt class="docutils literal"><span class="pre"></span></tt>). A simple example CGI callback script that
sends an e-mail is available in
<tt class="docutils literal"><span class="pre">examples/push-cgi/notifier/push-subscriber.cgi</span></tt> in the pony-build
source distribution.</p>
<p>Note that there are also utility functions in <tt class="docutils literal"><span class="pre">pony_build.rss</span></tt> for
helping to create RSS2 feeds and notify pubsubhubbub servers of
new results</p>
<div class="section">
<h1><a id="some-medium-term-ideas" name="some-medium-term-ideas">Some medium-term ideas</a></h1>
<p>One an initial release is out &amp; any obvious bugs are cleaned up, here
are some ideas for the next release.</p>
<p>A flexible view builder-and-saver would be nice; maybe in Django?
Think, &quot;separate results on this tag, etc; sort by time received;
expect these results to be shown or give an error.&quot;</p>
<p>It would be nice to be able to say &quot;I <em>expect</em> a result from this
buildhost, where is it!?&quot;</p>
<p>The build client 'subprocess' calls should be able to mimic 'tee',
that is, give real-time output of the build.</p>
<p>Some form of authentication from build clients. Josh Williams
suggests an approved client list (server side info about what clients
can conenect); I'd been thinking about a buildbot-style password setup,
where build clients shared a secret with the server. Both ideas are
good, I think.</p>
<p>In combination with authentication, we should put a default cap on the
total amount of data that can be dumped by an unauthenticated client.
Otherwise warez sites will be hosted inside of pony-build ;)</p>
<div class="section">
<h1><a id="development" name="development">Development</a></h1>
<p>pony-build is hosted on github, at: <a class="reference" href=""></a></p>
<p>To run the server tests:</p>
<pre class="literal-block">
python -m
<p>To run the client tests:</p>
<pre class="literal-block">
cd client
<div class="section">
<h1><a id="design-and-ideas-for-the-future" name="design-and-ideas-for-the-future">Design and Ideas for the Future</a></h1>
<div class="section">
<h2><a id="ideas-that-are-easy-to-implement" name="ideas-that-are-easy-to-implement">Ideas that are easy to implement</a></h2>
<p>Build virtualenv in on the client side (as a Context?)</p>
<p>Dependency chains example on client side.</p>
<p>Integration with unittest, nose, py.test -&gt; ship results back to
central server.</p>
<div class="section">
<h2><a id="cleanup" name="cleanup">Cleanup</a></h2>
<p>Figure out a proper database abstraction, maybe?</p>
<p>Tests, duh.</p>
<div class="section">
<h2><a id="things-i-don-t-know-how-to-do" name="things-i-don-t-know-how-to-do">Things I don't know how to do...</a></h2>
<p>...and don't want to spend time learning ;)</p>
<p>Josh Williams suggests supporting something other than a wsgiref
server. I'm not sure this is really needed -- you can run whatever
WSGI app you want inside the server -- but I can see that it would
make things more flexible for people with existing Web server setups.
I think the way to do this is to make pony-build's (XML-RPC + WSGI
app) interace look like its own WSGI app, rather than hacking
SimpleXMLRPCServer and wsgiref together in an unholy union. Ping me
for details if you dare.</p>
<p>Seriously, check out both pony_build.server.PonyBuildServer and
pony_build.server.RequestHandler (the latter is the most interesting).</p>
<div class="section">
<h2><a id="some-general-design-principles" name="some-general-design-principles">Some general design principles</a></h2>
<p>Titus says: A number of people are interested in pony-build, and I've
gotten many suggestions already. This has basically forced me to
articulate a number of my design principles, including some that were
made un- or subconsciously, and/or just enshrined in my prototype
code. I may change some of these decisions for v2, but I'd just as
soon let buildbot pick up the higher-end ideas if they're game, too.</p>
<li><p class="first">All client/server interactions should be via RPC, and hence
transactional. No always-on connections, no real-time control by
the central server.</p>
<p>This is a major simplification and makes it possible to keep the
code base small and simple. Yay.</p>
<li><p class="first">No partial results. Doug Phillips ('dgou') suggested that we allow
build clients to &quot;push&quot; individual results as they happen, rather
than all at once at the end. I can't think of a good, simple way
to do that, and it's part of the 20% that I don't yet need myself.</p>
<p>Here's a proposal that I think would work, from Doug:</p>
<pre class="literal-block">
send &quot;create new record, marked unfinished&quot;
receive &quot;record marker, update token&quot;
send &quot;first results, authenticate with update token&quot;
receive &quot;ack&quot;
send &quot;2nd results, authenticate with update token&quot;
receive &quot;ack&quot;
send &quot;final results, authenticate with update token&quot;
receive &quot;ack&quot;
<div class="section">
<h2><a id="contributors" name="contributors">Contributors</a></h2>
<p>Jacob Kaplan-Moss, Max Laite, Jack Carlson, Fatima Cherkaoui, and Khushboo
Shakya have all contributed code and ideas.</p>
<p>(If I'm missing anyone, please drop me a note!)</p>
<div class="section">
<h2><a id="acks" name="acks">Acks</a></h2>
<p>Titus says: Jesse Noller, Doug Philips, and Josh Williams discussed
things with me and are, collectively, entirely responsible for any bad
design decisions; the good ones are all mine.</p>
<p>Seriously, I appreciate the suggestions and comments from these fine
people, even though Doug has been a jerk to me since then.</p>
<p>Eric Henry built what I would consider 'pony-build prototype 1',
<p>You can also read this discussion starting here,</p>
<a class="reference" href=""></a></blockquote>
<p>where Kumar suggests that I just use Hudson for chrissakes. He's
probably right.</p>
<p>Eric Holscher and Jacob Kaplan-Moss took the pony-build idea and ran
with it, producing a parallel universe of Django-based reporting
servers and REST-ish clients that report via JSON. Check out and 'pony_barn' to see their approach in action.</p>
<div class="section">
<h2><a id="references" name="references">References</a></h2>
<p>webhooks: <a class="reference" href=""></a></p>
<p>CTB 2/24/10</p>