• Automatically queue Buildbot master restarts

    Buildbot comes with built-in functionality for clean restarts,
    which entail starting a new Buildbot process that does the following:
    - Cleanly shut down the existing buildmaster (existing instance)
      by waiting for pending builds to be finshed,
      ending the existing process.
    - Start a new buildmaster in the new process, taking over.
    
    Note that the new process becomes the new daemon,
    and thus needs to linger/be kept alive and managed.
    
    Upstart's built-in restart functionality hard-kills the existing
    process, which is undesirable; it's also hard to make Upstart wait
    for pending builds before stopping the existing process,
    as only Buildbot knows about any pending builds.
    Additionally, Buildbot has a limited reload functionality,
    but there are many pitfalls, gotchas, and inconsistencies,
    and it is not recommended for customized installations like ours.
    (e.g. re-loading imports doesn't work.)
    
    Note that when a Buildbot clean restart is requested, there are multiple
    processes running simultaneously.
    Model this in Upstart by using an "instance" job,
    which makes it easy to queue restarts and monitor them,
    without having to leave processes running in e.g. tmux or screen.
    
    Add an additional Upstart task to spawn a single instance of the
    Buildbot master service at boot-up,
    or during a Salt deploy if no instances are alive.
    More instances of the buildbot-master job can be started manually,
    or automatically by Salt, to queue up restarts.
    (The original process will exit gracefully after some time.)
    All instances will stop gracefully on shutdown.
    
    Note that the switch to an instance job means each separate instance
    must be differentiated by a `reason` variable, which is not used for
    any other purpose; this is automatically set to the current date/time,
    suffixed with some metadata about the reason for the instance.
    This has a side effect of creating separate Upstart logs
    (in `/var/log/upstart` for each instance), but the choice of reason
    keeps the logs sorted by date.
    
    Finally, automate this by having Salt queue a new restart job instance
    if anything about the Buildbot master configuration (or package)
    changes.
    aneeshusa committed Dec 27, 2016