Merge branch '9/sourcestamps' into nine

Conflicts:
	README.md
	master/buildbot/db/sourcestampsets.py

README.md

@@ -66,15 +66,25 @@ At this point, most of the configuration objects (masters, schedulers, builders,
 As data that has typically been in build pickles is migrated to the DB, all existing status code (web status, mail notifier, IRC, etc.) will become increasingly broken, because this code uses synchronous calls to get status information.
 Each of these status displays will need to be rewritten based on the Data API.
 
-## Infrastructure ##
+## Process ##
 
-* Optimize the type verification system by dynamically creating verifier functions once.
+The "process" part of Buildbot is the part that coordinates all of the other parts - the master, botmaster, etc.
+
+* request merging (or queue collapsing) - see https://plus.google.com/105883044168332773236/posts/TG8DHus4L4D
 
 ## Documentation ##
 
+* Document naming conventions (below) in the style guide, make sure everything adheres
 * Put update methods in the appropriate resource-type files, rather than in ``data.rst``
 * Move data API how-to guides into a separate file
 
+### Naming Conventions ###
+
+* interCaps for Python
+* interCaps for JS?
+* data id handling: object has `id`; foreign keys use `{fooid}`, with `foo` sometimes shortened; blah blah
+* DB API id handling (similar)
+
 ## Data API ##
 
 ### Tests ###
@@ -99,13 +109,12 @@ It's safe to leave tasks that have significant prerequisites - particularly the
 The outstanding resource types are:
 
 * scheduler (underlying DB API complete)
-* builder
-* sourcestamp
 * buildrequest
 * build
 * step
 * logfile
 * buildslave
+* changesource
 
 #### Schedulers ####
 
@@ -117,16 +126,17 @@ When a master becomes inactive (either on its own, or having failed its heartbea
 
 It should be possible to list the master on which a scheduler is running at e.g., `/scheduler/:schedulerid/master`
 
-#### Builders ####
-
-* Add identifiers for builders (url-component safe strings), and allow use of those in place of integer IDs in data API paths.
-
 #### Buildrequests ####
 
 Buildrequests include a `builderid` field which will need to be determined from the builders table.
 It should probably be an error to schedule a build on a builder that does not exist, as that build will not be executed.
 It's OK to schedule a build on a builder that's not implemented by a running master, though.
 
+Tasks:
+
+* define message formats, potentially including `claimed_at` and `masterid` in the "claimed" action
+* verify docs match validator
+
 ### Other Resource-Type Related Tasks ###
 
 * ``addBuildset`` currently sends messages about buildrequests directly.
@@ -135,14 +145,26 @@ It's OK to schedule a build on a builder that's not implemented by a running mas
 
 ### Misc Data API Work ###
 
+* addBuildset should take a list of builder IDs (involves integrating IDs into process Builder objects)
 * Paging, filtering, and so on of data API results.
 * Parsing of endpoint options is currently left to the endpoint, which will lead to inconsistencies.
   Add and document some helper methods to ``base.Endpoint`` for parsing e.g., boolean options (supporting on/off, 0/1, true/false, etc.)
 * If several rtypes have `_db2data` functions or similar (so far masters and changes both do), then make that an idiom and document it.
+* Move the methods of BuilderControl to update methods of the Builder resouce type (or other places as appropriate), and add control methods where appropriate.
+  In particular, implement `rebuildBuild` properly.
+* Use DateTimes everywhere
 
 ## Status Rewrites ##
 
-The following will need to be rewritten
+The following will need to be rewritten:
+
+* IRC (words.py)
+* MailNotifier
+* StatusClient (maybe)
+* WebStatus (already in progress with buildbot-www)
+* Tinderbox (maybe)
+* `status_gerrit`
+* `status_push` (maybe)
 
 ## Web ##
 
@@ -173,6 +195,11 @@ Testing javascript and json api interaction is tricky. Few design principles:
   - Need a js test mode that has to be enabled in master.cfg, in order to prevent prod's db to be corrupted by tests if malicious people launch them
   - Test mode will add some data api to inject pre-crafted events in the data flow.
 
+### REST API ###
+
+* Use plurals in path elements (/changes/NNN rather than /change/NNN)
+* Return dates as strings
+
 ### Vague Ideas ###
 
 These are just "things that need doing", where we don't have much idea *how* yet:
@@ -188,19 +215,12 @@ These are just "things that need doing", where we don't have much idea *how* yet
 
 ### Schema Changes ###
 
-  * Remove ``is_dir`` from the changes table (and ignore/remove it everywhere else)
-  * ``changesources``
-
-    * New table
-    * Migration script + tests
-    * DB API module + docs, type verifier, and interface tests
-    * Fake implementation that passes unit tests 
-    * Add TODO for data API implementation
-
-  * replace ``builds`` with a useful representation of builds
-  * ``steps``
-  * ``logs``
-  * ``logchunks``
+* Remove ``is_dir`` from the changes table (and ignore/remove it everywhere else)
+* ``changesources``
+* replace ``builds`` with a useful representation of builds
+* ``steps``
+* ``logs``
+* ``logchunks``
 
 ### DB API Changes ###
 
@@ -214,10 +234,12 @@ These are just "things that need doing", where we don't have much idea *how* yet
 ### Documentation ###
 
 * Document Buildbot's behavior for a DBA: isolation assumptions, dependencies on autogenerated IDs, read-after-write expectations, buildrequest claiming, buildset completion detection
+* Document how to handle merging of build requests now that SourceStamps are dead.
 
 ### Miscellaneous ###
 
 * Look carefully at race conditions around masters being marked inactive
+* Now that schedulers don't give lists of changes to addBuildset, there's no need to keep a list of important/unimportant changes; just state.
 
 ## Later ##
 
@@ -231,6 +253,11 @@ The MQ layer currently only has a simple (single-master) implementation.  We sho
 * AMP-based master-to-master communication (full mesh, with every master talking to every other master)
 * ZeroMQ
 
+### Data API ###
+
+* Add identifiers for builders (url-component safe strings), and allow use of those in place of integer IDs in data API paths.
+  For example, external users might find it easier to look at a request queue with `/builder/smoketest/requests` rather than `/buidler/12/requests`.
+
 ### Infrastructure ###
 
 * Use some fancy algorithms to delay message transmission until the data is known-good in the database, for cases where the underlying database is asynchronously replicated.

master/buildbot/changes/changes.py

@@ -15,11 +15,10 @@
 
 from __future__ import with_statement
 
-import os, time
-from cPickle import dump
+import time
 
 from zope.interface import implements
-from twisted.python import log, runtime
+from twisted.python import log
 from twisted.internet import defer
 from twisted.web import html
 from buildbot.util import datetime2epoch
@@ -203,73 +202,3 @@ def getLogs(self):
         return {}
 
 
-class ChangeMaster: # pragma: no cover
-    # this is a stub, retained to allow the "buildbot upgrade-master" tool to
-    # read old changes.pck pickle files and convert their contents into the
-    # new database format. This is only instantiated by that tool, or by
-    # test_db.py which tests that tool. The functionality that previously
-    # lived here has been moved into buildbot.changes.manager.ChangeManager
-
-    def __init__(self):
-        self.changes = []
-        # self.basedir must be filled in by the parent
-        self.nextNumber = 1
-
-    def saveYourself(self):
-        filename = os.path.join(self.basedir, "changes.pck")
-        tmpfilename = filename + ".tmp"
-        try:
-            with open(tmpfilename, "wb") as f:
-                dump(self, f)
-            if runtime.platformType  == 'win32':
-                # windows cannot rename a file on top of an existing one
-                if os.path.exists(filename):
-                    os.unlink(filename)
-            os.rename(tmpfilename, filename)
-        except Exception:
-            log.msg("unable to save changes")
-            log.err()
-
-    # This method is used by contrib/fix_changes_pickle_encoding.py to recode all
-    # bytestrings in an old changes.pck into unicode strings
-    def recode_changes(self, old_encoding, quiet=False):
-        """Processes the list of changes, with the change attributes re-encoded
-        unicode objects"""
-        nconvert = 0
-        for c in self.changes:
-            # give revision special handling, in case it is an integer
-            if isinstance(c.revision, int):
-                c.revision = unicode(c.revision)
-
-            for attr in ("who", "comments", "revlink", "category", "branch", "revision"):
-                a = getattr(c, attr)
-                if isinstance(a, str):
-                    try:
-                        setattr(c, attr, a.decode(old_encoding))
-                        nconvert += 1
-                    except UnicodeDecodeError:
-                        raise UnicodeError("Error decoding %s of change #%s as %s:\n%r" %
-                                        (attr, c.number, old_encoding, a))
-
-            # filenames are a special case, but in general they'll have the same encoding
-            # as everything else on a system.  If not, well, hack this script to do your
-            # import!
-            newfiles = []
-            for filename in util.flatten(c.files):
-                if isinstance(filename, str):
-                    try:
-                        filename = filename.decode(old_encoding)
-                        nconvert += 1
-                    except UnicodeDecodeError:
-                        raise UnicodeError("Error decoding filename '%s' of change #%s as %s:\n%r" %
-                                        (filename.decode('ascii', 'replace'),
-                                         c.number, old_encoding, a))
-                newfiles.append(filename)
-            c.files = newfiles
-        if not quiet:
-            print "converted %d strings" % nconvert
-
-class OldChangeMaster(ChangeMaster): # pragma: no cover
-    # this is a reminder that the ChangeMaster class is old
-    pass
-# vim: set ts=4 sts=4 sw=4 et:

master/buildbot/clients/debug.py

@@ -50,7 +50,6 @@ def __init__(self, master="localhost:8007", passwd="debugpw"):
         c('do_reload', self.do_reload)
         c('do_rebuild', self.do_rebuild)
         c('do_poke_irc', self.do_poke_irc)
-        c('do_build', self.do_build)
         c('do_ping', self.do_ping)
         c('do_commit', self.do_commit)
         c('on_usebranch_toggled', self.usebranch_toggled)
@@ -104,26 +103,6 @@ def do_poke_irc(self, widget):
         d = self.remote.callRemote("pokeIRC")
         d.addErrback(self.err)
 
-    def do_build(self, widget):
-        if not self.remote:
-            return
-        name = self.buildname.get_text()
-        branch = None
-        if self.xml.get_widget("usebranch").get_active():
-            branch = self.xml.get_widget('branch').get_text()
-            if branch == '':
-                branch = None
-        revision = None
-        if self.xml.get_widget("userevision").get_active():
-            revision = self.xml.get_widget('revision').get_text()
-            if revision == '':
-                revision = None
-        reason = "debugclient 'Request Build' button pushed"
-        properties = {}
-        d = self.remote.callRemote("requestBuild",
-                                   name, reason, branch, revision, properties)
-        d.addErrback(self.err)
-
     def do_ping(self, widget):
         if not self.remote:
             return

master/buildbot/clients/tryclient.py

@@ -32,11 +32,16 @@
 from twisted.python.procutils import which
 from twisted.spread import pb
 
-from buildbot.sourcestamp import SourceStamp
 from buildbot.status import builder
 from buildbot.util import json
 from buildbot.util import now
 
+class SourceStamp(object):
+
+    def __init__(self, branch, revision, patch, repository=''):
+        self.branch = branch
+        self.revision = revision
+        self.patch = patch
 
 class SourceStampExtractor: