wait / block plugin for others builds or jobs

[bradrydzewski]

I've been kicking around the idea of a wait or block plugin. This would block the build until some sort of event happens related to other builds or jobs (matrix). These are some example use cases:

• wait until all other jobs in the matrix complete. This is useful when you have (let's say) a matrix build with 10 jobs and you want job 10 to deploy once the other 9 jobs are complete
• wait until all previously started builds complete before moving forward. This is useful when you have builds running concurrently and you don't want multiple deployments to execute concurrently

The initial implementation will need to be a plugin, which means it would be configured as a step in your pipeline just like any other Drone plugin. The plugin could use the Drone API to poll the server, blocking the pipeline until conditions are met.

This is an example of what the plugin could look like:

pipeline:
  build:
     ...
  test:
    ...
  wait:
    image: plugin/wait
    timeout: 10m
    for: [ 1.5, 1.6 ]
    where:
      environment:
        golang: 1.7

matrix:
  golang: [ 1.5, 1.6, 1.7 ]

The reason for building this as a plugin and not directly into Drone core is that teams will have different requirements for blocking and waiting. Our policy is pretty consistent that these sort of features need to be created as external plugins first, before we would consider adding the feature to Drone core.

I also want to set expectations that I have no plans to build this plugin myself. It is not a feature I personally need, and since anyone can build and publish their own plugin independently, there is no reason someone else can't build and publish to the plugin index. To learn more about creating plugins see http://readme.drone.io/plugins/creating-custom-plugins-bash/

[gtaylor]

I think something like this could be helpful, but the biggest thing I'll be looking to do in the coming era of the re-worked yaml is write a plugin that waits until service containers are fully started. So in a case like that, the most important thing for that usage case is:

• The ability to add an arbitrary container before the build step that can use my language/toolset of choice.

This is not really what you are talking about above, but I wanted to make sure that the other side of this was considered.

[bradrydzewski]

I updated the issue title and description to be a bit more specific. I'm not looking for a generic wait plugin, but instead looking for something that prevents this situation:

How can I define that only a single deploy can run at a time. For example we merge multiple things into master quickly, this fires multiple deploys that run in parallel.

[thomasf]

I also want the pre build plugin stuff, A couple of my builds needs to fetch some external resources before starting which would be great.

Maybe, just maybe it would be good to have a way to specify this as a graph via some sorts of tags?

(I just reused the wait keyword from above, there are probably better names for the config format)

build:
  wait:
    after:
     - pgup
     - getassets

compose:
  database:
    image: postgres

  pgup:
    image: pg-up-watcher
    wait:
      name: pgup
      timeout: 10m

  getassets:
    image: my-s3-downloader
    wait:
      name: getassets

reusing a "name":

build:
  commands:
    - ./ci.sh runci
  wait:
    after: build-ready

compose:
  database:
    image: postgres

  pgup:
    image: pg-up-watcher
    wait:
      name: build-ready
      timeout: 10m

  getassets:
    image: my-s3-downloader
    wait:
      name: build-ready

a dotted notation could be used without naming things:

build:
  commands:
    - ./ci.sh runci
  wait:
    after: compose.pgup

compose:
  database:
    image: postgres

  pgup:
    image: pg-up-watcher

I don't know if this falls under the crazy flexible ultra-configurable queue system category or not :) As long as it's can be expressed as DAG the execution planning should be that complicated, right?. IIIRC Tarjans algorithm can be used to resolve the dependecy order easily for this use case..

addition: the name: section key should probably not be inside the wait: block.. It's more likley that it's better to have them at the plugin level and call it labels or something that does to conflict with other features. The rationale is that named sections can be useful for other purposes than waiting.

[constantm]

Any new thoughts on this? Things we could really use is limits per branch, and as you mentioned, limits for running deploys. Would love to help on the implementation but I'm going to have to learn Go first. :D

[bradrydzewski]

@thomasf while I appreciate you taking the time to provide feedback, I do not think this solves the problem described in this issue. This issue focuses on waiting for other builds and jobs (external to the current pipeline) to complete. Based on your comments above you are looking for a way to wait for other containers within the same pipeline to complete. I don't think we need any special yaml notation for this, at least not as of 0.5

Would love to help on the implementation but I'm going to have to learn Go first

Note that since this initial implementation needs to be a plugin, there is no need to learn Go. Plugins can be written in any language, including bash. Check out http://readme.drone.io/plugins/creating-custom-plugins-bash/

[ozbillwang]

my user case is, I need to run terraform plan (dry-run, similar as ansible option --check) first, block/pause this pipeline, review the logs, if everything goes well, I'd like to click a confirm button or input y/n to run next step terraform apply to apply the changes or give up.

Another user case is, I'd like to block deployment if environment is production. So I can make sure the production changes are implemented in planned time.

I review the requests above, seems we are thinking about different purposes and there are too complex to be implemented at one shot. Can we start from a simple one first?

Just one feature, if step is block, wait for click.

pipeline:
  wait:
    image: plugin/wait
    timeout: 10m  # (or -1) wait forever

or simpler (any pipeline named wait will be the block step)

pipeline:
  wait:

I prefer to have this feature in drone server core directly, more than to be implemented with plugin.

The only concern is, if in wait process, will the agent to be hold that no new job can be deployed by this agent? Can we release the agent when in block process?

[tonglil]

wait until all previously started builds complete before moving forward. This is useful when you have builds running concurrently and you don't want multiple deployments to execute concurrently

This would be really useful. Currently we're planning to use/build an external system for locking.

[bradrydzewski]

I am going to close this for a few reasons ...

First is this issues provides a design for a plugin (that anyone is welcome to build) to enable blocking and locking (below). I recommend using something like redis as the backend for such a plugin, since it has these semantics built-in.

pipeline:
  wait:
    image: plugin/wait
    timeout: 10m  # (or -1) wait forever

this solves one use case documented above

wait until all other jobs in the matrix complete

Second we have an open issue for limiting the number of concurrent builds per-repository. This could be used an alternate to the above solution to throttle. See #758

this solves two use cases documented above

wait until all previously started builds complete before moving forward.
limits per branch [and] limits for running deploys

Thrid matrix builds will likely change substantially once we have fan-in and fan-out enabled. It will effectively allow you to fan-out to multiple machines, test muliptle matrixes, and fan-in.

this solves one use case documented above

wait until all other jobs in the matrix complete

Fourth we have an open issue for blocking the pipeline pending manual intervention at #2126. This solves the use case mentioned by @ozbillwang

pause the pipeline and click a confirm button to proceed

[bradrydzewski]

I am locking this issue, since we have more specific issues that target each of these use cases. If you have any further comments, please direct them to the relevant issue mentioned above.