• Upgrade Buildbot master DB on upgrade

    When upgrading the Buildbot master version,
    or starting from a fresh deploy (no existing database),
    the Buildbot database must be upgraded in order
    for Buildbot to start normally.
    (The DB is usually created during `buildbot create-master`,
    but we want to avoid checking in the database.)
    Add a `cmd.run` state that only runs if the Buildbot version is changed,
    using an `onchanges` requisite.
    
    The upgrade script requires that Buildbot is not running,
    presuambly to avoid conflicting updates,
    so Buildbot must be stopped before running the upgrade.
    We want to perform a clean stop, but the built-in `buildbot stop`
    command is nonblocking and will return immediately, without waiting for
    the existing Buildbot instance to finish.
    Buildbot has blocking/waiting for stop functionality built-in but not
    exposed, so add a small helper script to stop Buildbot and block until
    it is finished shutting down, and invoke it before the upgrade.
    
    An alternative would be trying to use Upstart or Salt directly
    to stop Buildbot, as a clean shutdown boils down to sending a SIGUSR1
    to the Buildbot process (only if one is running), in the Unix tradition.
    However, this would be hard to integrate with; in particular, we need to
    wait for the existing Buildbot process to finish running; the easiest
    way to integrate this into a Salt state (without writing a custom Salt
    state) is to start a process to do the waiting, hence the stop script.
    
    Note that the upgrade-master command also adds various other cruft
    to the master directory; the Buildbot internal upgradeDatabase API
    is not called because it is layered in Twisted Reactor/inline callback
    goop, and it is simpler to just call the CLI command.
    
    Also update the Buildbot master states to be more strict
    about using requisites for better ordering control,
    and re-order/space out states for a better reading flow.
    aneeshusa committed Dec 27, 2016