Skip to content

cherkf/pony-build

BranchesTags
 
 

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

357 Commits

bin

bin

 
 

client

client

 
 

examples

examples

 
 

pony_build

pony_build

 
 

.gitignore

.gitignore

 
 

IDEAS

IDEAS

 
 

README.html

README.html

 
 

README.txt

README.txt

 
 

setup.py

setup.py

 
 

Repository files navigation

<?xml version="1.0" encoding="utf-8" ?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
<meta name="generator" content="Docutils 0.4: http://docutils.sourceforge.net/" />
<title>pony-build</title>
<style type="text/css">

/*
:Author: David Goodger
:Contact: goodger@users.sourceforge.net
:Date: $Date: 2005-12-18 01:56:14 +0100 (Sun, 18 Dec 2005) $
:Revision: $Revision: 4224 $
:Copyright: This stylesheet has been placed in the public domain.

Default cascading style sheet for the HTML output of Docutils.

See http://docutils.sf.net/docs/howto/html-stylesheets.html for how to
customize this style sheet.
*/

/* used to remove borders from tables and images */
.borderless, table.borderless td, table.borderless th {
  border: 0 }

table.borderless td, table.borderless th {
  /* Override padding for "table.docutils td" with "! important".
     The right padding separates the table cells. */
  padding: 0 0.5em 0 0 ! important }

.first {
  /* Override more specific margin styles with "! important". */
  margin-top: 0 ! important }

.last, .with-subtitle {
  margin-bottom: 0 ! important }

.hidden {
  display: none }

a.toc-backref {
  text-decoration: none ;
  color: black }

blockquote.epigraph {
  margin: 2em 5em ; }

dl.docutils dd {
  margin-bottom: 0.5em }

/* Uncomment (and remove this text!) to get bold-faced definition list terms
dl.docutils dt {
  font-weight: bold }
*/

div.abstract {
  margin: 2em 5em }

div.abstract p.topic-title {
  font-weight: bold ;
  text-align: center }

div.admonition, div.attention, div.caution, div.danger, div.error,
div.hint, div.important, div.note, div.tip, div.warning {
  margin: 2em ;
  border: medium outset ;
  padding: 1em }

div.admonition p.admonition-title, div.hint p.admonition-title,
div.important p.admonition-title, div.note p.admonition-title,
div.tip p.admonition-title {
  font-weight: bold ;
  font-family: sans-serif }

div.attention p.admonition-title, div.caution p.admonition-title,
div.danger p.admonition-title, div.error p.admonition-title,
div.warning p.admonition-title {
  color: red ;
  font-weight: bold ;
  font-family: sans-serif }

/* Uncomment (and remove this text!) to get reduced vertical space in
   compound paragraphs.
div.compound .compound-first, div.compound .compound-middle {
  margin-bottom: 0.5em }

div.compound .compound-last, div.compound .compound-middle {
  margin-top: 0.5em }
*/

div.dedication {
  margin: 2em 5em ;
  text-align: center ;
  font-style: italic }

div.dedication p.topic-title {
  font-weight: bold ;
  font-style: normal }

div.figure {
  margin-left: 2em ;
  margin-right: 2em }

div.footer, div.header {
  clear: both;
  font-size: smaller }

div.line-block {
  display: block ;
  margin-top: 1em ;
  margin-bottom: 1em }

div.line-block div.line-block {
  margin-top: 0 ;
  margin-bottom: 0 ;
  margin-left: 1.5em }

div.sidebar {
  margin-left: 1em ;
  border: medium outset ;
  padding: 1em ;
  background-color: #ffffee ;
  width: 40% ;
  float: right ;
  clear: right }

div.sidebar p.rubric {
  font-family: sans-serif ;
  font-size: medium }

div.system-messages {
  margin: 5em }

div.system-messages h1 {
  color: red }

div.system-message {
  border: medium outset ;
  padding: 1em }

div.system-message p.system-message-title {
  color: red ;
  font-weight: bold }

div.topic {
  margin: 2em }

h1.section-subtitle, h2.section-subtitle, h3.section-subtitle,
h4.section-subtitle, h5.section-subtitle, h6.section-subtitle {
  margin-top: 0.4em }

h1.title {
  text-align: center }

h2.subtitle {
  text-align: center }

hr.docutils {
  width: 75% }

img.align-left {
  clear: left }

img.align-right {
  clear: right }

ol.simple, ul.simple {
  margin-bottom: 1em }

ol.arabic {
  list-style: decimal }

ol.loweralpha {
  list-style: lower-alpha }

ol.upperalpha {
  list-style: upper-alpha }

ol.lowerroman {
  list-style: lower-roman }

ol.upperroman {
  list-style: upper-roman }

p.attribution {
  text-align: right ;
  margin-left: 50% }

p.caption {
  font-style: italic }

p.credits {
  font-style: italic ;
  font-size: smaller }

p.label {
  white-space: nowrap }

p.rubric {
  font-weight: bold ;
  font-size: larger ;
  color: maroon ;
  text-align: center }

p.sidebar-title {
  font-family: sans-serif ;
  font-weight: bold ;
  font-size: larger }

p.sidebar-subtitle {
  font-family: sans-serif ;
  font-weight: bold }

p.topic-title {
  font-weight: bold }

pre.address {
  margin-bottom: 0 ;
  margin-top: 0 ;
  font-family: serif ;
  font-size: 100% }

pre.literal-block, pre.doctest-block {
  margin-left: 2em ;
  margin-right: 2em ;
  background-color: #eeeeee }

span.classifier {
  font-family: sans-serif ;
  font-style: oblique }

span.classifier-delimiter {
  font-family: sans-serif ;
  font-weight: bold }

span.interpreted {
  font-family: sans-serif }

span.option {
  white-space: nowrap }

span.pre {
  white-space: pre }

span.problematic {
  color: red }

span.section-subtitle {
  /* font-size relative to parent (h1..h6 element) */
  font-size: 80% }

table.citation {
  border-left: solid 1px gray;
  margin-left: 1px }

table.docinfo {
  margin: 2em 4em }

table.docutils {
  margin-top: 0.5em ;
  margin-bottom: 0.5em }

table.footnote {
  border-left: solid 1px black;
  margin-left: 1px }

table.docutils td, table.docutils th,
table.docinfo td, table.docinfo th {
  padding-left: 0.5em ;
  padding-right: 0.5em ;
  vertical-align: top }

table.docutils th.field-name, table.docinfo th.docinfo-name {
  font-weight: bold ;
  text-align: left ;
  white-space: nowrap ;
  padding-left: 0 }

h1 tt.docutils, h2 tt.docutils, h3 tt.docutils,
h4 tt.docutils, h5 tt.docutils, h6 tt.docutils {
  font-size: 100% }

tt.docutils {
  background-color: #eeeeee }

ul.auto-toc {
  list-style-type: none }

</style>
</head>
<body>
<div class="document" id="pony-build">
<h1 class="title">pony-build</h1>
<p>pony-build is a simple continuous integration package that lets you
run a server to display client build results.  It consists of two
components, a server (which is run in some central &amp; accessible
location), and one or more clients (which must be able to contact the
server via HTTP).</p>
<p>Philosophy statement: good development tools for Python should be easy
to install, easy to hack, and not overly constraining.  Two out of
three ain't bad ;).</p>
<p>The pony-build architectural model is this: use decoupled components
that communicate via webhooks whenever possible.</p>
<p>Also see <a class="reference" href="http://buildbot.sf.net/">buildbot</a>.</p>
<div class="section">
<h1><a id="links" name="links">Links</a></h1>
<p>pony-build is hosted on github.</p>
<p>pony-build central repository: <a class="reference" href="http://github.com/ctb/pony-build">http://github.com/ctb/pony-build</a></p>
<p>pony-build issue tracking: <a class="reference" href="http://github.com/ctb/pony-build/issues">http://github.com/ctb/pony-build/issues</a></p>
<p>pony-build wiki: <a class="reference" href="http://wiki.github.com/ctb/pony-build">http://wiki.github.com/ctb/pony-build</a></p>
<p>pony-build mailing list: <a class="reference" href="http://lists.idyll.org/listinfo/pony-build">http://lists.idyll.org/listinfo/pony-build</a></p>
</div>
<div class="section">
<h1><a id="requirements" name="requirements">Requirements</a></h1>
<dl class="docutils">
<dt>Server side:</dt>
<dd><p class="first">Requires Python 2.6 or above.</p>
<p>Jinja2 (easy_installable).</p>
<p class="last">For the Quixote Web UI, Quixote 2.6 (also easy_installable).</p>
</dd>
<dt>Client side:</dt>
<dd>Python.  Should work down to Python 2.4.  Developed with 2.5.</dd>
</dl>
</div>
<div class="section">
<h1><a id="pony-build-server" name="pony-build-server">pony-build server</a></h1>
<p>The command:</p>
<pre class="literal-block">
python -m pony_build.web.run &lt;shelve filename&gt; -p &lt;port&gt;
</pre>
<p>will run the Quixote-based pony-build Web app on the given port,
reading &amp; writing from the sqlite database in 'filename'.</p>
<p>For example,</p>
<pre class="literal-block">
python -m pony_build.web.run test.db -p 8080
</pre>
<p>will run a server that can be accessed on <a class="reference" href="http://localhost:8080/">http://localhost:8080/</a>.  This
server will report on whatever results are sent to it by the client (see
below), based on the package name ('name', below).</p>
<p>See 'architecture, and extending pony-build', below.</p>
</div>
<div class="section">
<h1><a id="pony-build-client-scripts" name="pony-build-client-scripts">pony-build client scripts</a></h1>
<div class="section">
<h2><a id="build-scripts" name="build-scripts">Build scripts</a></h2>
<p>Client build scripts are just scripts that set up &amp; run a list of commands:</p>
<pre class="literal-block">
from pony_client import BuildCommand, TestCommand, do, send

name = 'example'
server_url = 'http://localhost:8080/xmlrpc'

commands = [ BuildCommand(['/bin/echo', 'hello, world'], name='step 1'),
             TestCommand(['/bin/echo', 'this is a test'], name='step 2')
             ]

results = do('package', commands)
send('http://localhost:8080/xmlrpc', results)
</pre>
<p>Client results are communicated to the server by XML-RPC, so the client
must be able to reach the server via the HTTP protocol.</p>
<p>See <tt class="docutils literal"><span class="pre">client/build-cpython</span></tt> for an example of building a Subversion-based
C project (checkout, configure, make, run tests).</p>
<p>See <tt class="docutils literal"><span class="pre">client/build-pony-build</span></tt> for an example of building a Git-based
Python project (clone, build, run tests).</p>
<p>Note that 'pony_client' doesn't depend on the rest of pony-build, so you
can distribute it with other packages if you want.</p>
</div>
<div class="section">
<h2><a id="client-query-scripts" name="client-query-scripts">Client query scripts</a></h2>
<p>Client query scripts request information from the server.  For example,
see 'bin/notify-failure-email', which checks the status of the last build
of a particular package and sends an e-mail.</p>
</div>
</div>
<div class="section">
<h1><a id="architecture-and-extending-pony-build" name="architecture-and-extending-pony-build">Architecture, and extending pony-build</a></h1>
<p>The pony-build server is basically a storage receptacle for &quot;bags&quot; of
key-value pairs.  It's easiest point of extension is in the Web
interface, where you can substitute any WSGI app object to serve Web
pages; see 'bin/run-server', and the function call to
'server.create(...)' (aka pony_build.server.create(...)'.</p>
<p>To write a new Web UI, you will need access to the stored pony-build
server data.  You should work through the 'PonyBuildCoordinator'
interface in 'pony_build.coordinator'; you can get a handle to the
current coordinator object with 'pony_build.server.get_coordinator()'.
<strong>The coordinator API is will be stable and public</strong>, after suitable
evolution.</p>
<div class="section">
<h2><a id="client-to-server-communication" name="client-to-server-communication">Client-to-server communication</a></h2>
<p>Each client sends two bags of information: the first, 'client_info',
contains information global to the build client, such as package name,
host name, architecture, and a list of tags.  The second,
'results_list', is an ordered list of dictionaries, each one
representing a build step.  (The default contents of these dictionaries
are pretty obvious: status, stderr, stdout, etc.)  Upon receipt of
this info, the pony-build server creates a third object, a dictionary,
containing information such as the server time at which the result
was received,</p>
<p>These three bags of info -- 'receipt', 'client_info', and
'results_list' -- are it.  The coordinator functions give you ways to
slice and dice which results set you want (e.g. latest for a
particular package), and then usually return one or more triples of
(receipt, client_info, results_list).</p>
<p>Clients can send arbitrary key/value pairs in their &quot;bags&quot;; two
simple ways to extend things are to add new k/v pairs for specific
purposes, and/or to use the 'tags' key in the client_info dict.
The 'tags' associated value is a list of strings.</p>
<p>receipt['result_key'] is the internal key used to store the result.</p>
</div>
<div class="section">
<h2><a id="notifications" name="notifications">Notifications</a></h2>
<p>RSS2 and pubsubhubbub (<a class="reference" href="http://code.google.com/p/pubsubhubbub/">http://code.google.com/p/pubsubhubbub/</a>) are the
core of the notification system built into pony_build; the &quot;officially
correct&quot; way to actively notify interested people of build results is
to publish them via RSS2, push them to a pubsubhubbub server, and let
someone else deal with translating those into e-mail alerts, etc.</p>
<p>All of the RSS feeds that pony-build makes available can be posted to
pubsubhubbub with the proper configuration (see -P and -S options to
<tt class="docutils literal"><span class="pre">pony_build.web.run</span></tt>).  A simple example CGI callback script that
sends an e-mail is available in
<tt class="docutils literal"><span class="pre">examples/push-cgi/notifier/push-subscriber.cgi</span></tt> in the pony-build
source distribution.</p>
<p>Note that there are also utility functions in <tt class="docutils literal"><span class="pre">pony_build.rss</span></tt> for
helping to create RSS2 feeds and notify pubsubhubbub servers of
new results</p>
</div>
</div>
<div class="section">
<h1><a id="some-medium-term-ideas" name="some-medium-term-ideas">Some medium-term ideas</a></h1>
<p>One an initial release is out &amp; any obvious bugs are cleaned up, here
are some ideas for the next release.</p>
<p>A flexible view builder-and-saver would be nice; maybe in Django?
Think, &quot;separate results on this tag, etc; sort by time received;
expect these results to be shown or give an error.&quot;</p>
<p>It would be nice to be able to say &quot;I <em>expect</em> a result from this
buildhost, where is it!?&quot;</p>
<p>The build client 'subprocess' calls should be able to mimic 'tee',
that is, give real-time output of the build.</p>
<p>Some form of authentication from build clients.  Josh Williams
suggests an approved client list (server side info about what clients
can conenect); I'd been thinking about a buildbot-style password setup,
where build clients shared a secret with the server.  Both ideas are
good, I think.</p>
<p>In combination with authentication, we should put a default cap on the
total amount of data that can be dumped by an unauthenticated client.
Otherwise warez sites will be hosted inside of pony-build ;)</p>
</div>
<div class="section">
<h1><a id="development" name="development">Development</a></h1>
<p>pony-build is hosted on github, at: <a class="reference" href="http://github.com/ctb/pony-build">http://github.com/ctb/pony-build</a></p>
<p>To run the server tests:</p>
<pre class="literal-block">
python -m pony_build.tests.run
</pre>
<p>To run the client tests:</p>
<pre class="literal-block">
cd client
nosetests
</pre>
</div>
<div class="section">
<h1><a id="design-and-ideas-for-the-future" name="design-and-ideas-for-the-future">Design and Ideas for the Future</a></h1>
<div class="section">
<h2><a id="ideas-that-are-easy-to-implement" name="ideas-that-are-easy-to-implement">Ideas that are easy to implement</a></h2>
<p>Build virtualenv in on the client side (as a Context?)</p>
<p>Dependency chains example on client side.</p>
<p>Integration with unittest, nose, py.test -&gt; ship results back to
central server.</p>
</div>
<div class="section">
<h2><a id="cleanup" name="cleanup">Cleanup</a></h2>
<p>Figure out a proper database abstraction, maybe?</p>
<p>Tests, duh.</p>
</div>
<div class="section">
<h2><a id="things-i-don-t-know-how-to-do" name="things-i-don-t-know-how-to-do">Things I don't know how to do...</a></h2>
<p>...and don't want to spend time learning ;)</p>
<p>Josh Williams suggests supporting something other than a wsgiref
server.  I'm not sure this is really needed -- you can run whatever
WSGI app you want inside the server -- but I can see that it would
make things more flexible for people with existing Web server setups.
I think the way to do this is to make pony-build's (XML-RPC + WSGI
app) interace look like its own WSGI app, rather than hacking
SimpleXMLRPCServer and wsgiref together in an unholy union.  Ping me
for details if you dare.</p>
<p>Seriously, check out both pony_build.server.PonyBuildServer and
pony_build.server.RequestHandler (the latter is the most interesting).</p>
</div>
<div class="section">
<h2><a id="some-general-design-principles" name="some-general-design-principles">Some general design principles</a></h2>
<p>Titus says: A number of people are interested in pony-build, and I've
gotten many suggestions already.  This has basically forced me to
articulate a number of my design principles, including some that were
made un- or subconsciously, and/or just enshrined in my prototype
code.  I may change some of these decisions for v2, but I'd just as
soon let buildbot pick up the higher-end ideas if they're game, too.</p>
<blockquote>
<ul>
<li><p class="first">All client/server interactions should be via RPC, and hence
transactional.  No always-on connections, no real-time control by
the central server.</p>
<p>This is a major simplification and makes it possible to keep the
code base small and simple.  Yay.</p>
</li>
<li><p class="first">No partial results.  Doug Phillips ('dgou') suggested that we allow
build clients to &quot;push&quot; individual results as they happen, rather
than all at once at the end.  I can't think of a good, simple way
to do that, and it's part of the 20% that I don't yet need myself.</p>
<p>Here's a proposal that I think would work, from Doug:</p>
<pre class="literal-block">
send &quot;create new record, marked unfinished&quot;
receive &quot;record marker, update token&quot;
send &quot;first results, authenticate with update token&quot;
receive &quot;ack&quot;
send &quot;2nd results, authenticate with update token&quot;
receive &quot;ack&quot;
...
send &quot;final results, authenticate with update token&quot;
receive &quot;ack&quot;
</pre>
</li>
</ul>
</blockquote>
</div>
<div class="section">
<h2><a id="contributors" name="contributors">Contributors</a></h2>
<p>Jacob Kaplan-Moss, Max Laite, Jack Carlson, Fatima Cherkaoui, and Khushboo
Shakya have all contributed code and ideas.</p>
<p>(If I'm missing anyone, please drop me a note!)</p>
</div>
<div class="section">
<h2><a id="acks" name="acks">Acks</a></h2>
<p>Titus says: Jesse Noller, Doug Philips, and Josh Williams discussed
things with me and are, collectively, entirely responsible for any bad
design decisions; the good ones are all mine.</p>
<p>Seriously, I appreciate the suggestions and comments from these fine
people, even though Doug has been a jerk to me since then.</p>
<p>Eric Henry built what I would consider 'pony-build prototype 1',
project-builder-pie.</p>
<p>You can also read this discussion starting here,</p>
<blockquote>
<a class="reference" href="http://lists.idyll.org/pipermail/testing-in-python/2009-March/001277.html">http://lists.idyll.org/pipermail/testing-in-python/2009-March/001277.html</a></blockquote>
<p>where Kumar suggests that I just use Hudson for chrissakes.  He's
probably right.</p>
<p>Eric Holscher and Jacob Kaplan-Moss took the pony-build idea and ran
with it, producing a parallel universe of Django-based reporting
servers and REST-ish clients that report via JSON.  Check out
devmason.com and 'pony_barn' to see their approach in action.</p>
</div>
<div class="section">
<h2><a id="references" name="references">References</a></h2>
<p>webhooks: <a class="reference" href="http://webhooks.pbworks.com/">http://webhooks.pbworks.com/</a></p>
<p>--</p>
<p>CTB 2/24/10</p>
</div>
</div>
</div>
</body>
</html>

About

A simple continuous integration system, written in Python.

wiki.github.com/ctb/pony-build

Resources

Readme
Activity

Stars

2 stars

Watchers

1 watching

Forks

1 fork
Report repository

Releases

No releases published

Packages

No packages published

Languages

  • Python 100.0%