Matrix Builds / Sub-Builds / Parallel Builds

[jpadilla]

I'm wondering how would a .drone.yml would look if you'd want to test with multiple versions of Python, like for example, using Tox?

[bradrydzewski]

good question. what do you think the .drone.yml should look like?

Here are the use cases I'd like to cover with sub builds:

• I want to test code against python2.7 and python3.3 and pypy
• I want to test code against three different docker images (same thing as above)
• I want to test code against python2.7 and redis2.6 and redis2.8
• I want to test code against python2.7, but I want to run 3 "sub builds" in parallel to reduce execution time

Here are some design questions:

• How do the above use cases impact deployments?
• If we have 3 sub builds which one should be responsible for executing the deployment?
• If we have 1 sub builds responsible for deployment, should it wait for the other 2 to pass?
• How do we represent all of this in the yaml
• How do we display the results in the UI (e.g. website design)

Although I think we can draw some inspiration from Travis I do not want to just copy their approach as I think it would limit our capabilities. With Travis your build is heavily tied to a language:

language: go

With Drone your build is tied to a Docker image. The image defines the environment. This may seem like a nuance, but this is really important. Drone doesn't care about language. Drone will never dictate which languages you can or cannot use. We need a yaml file that is re-imaged for Docker.

I consider this a high priority feature. Hopefully we can get a discussion started here and come up with some options.

[jpadilla]

@bradrydzewski great use cases here. Wasn't even thinking about those. One thing that comes to mind is having Docker images that contain all possible versions. For example, A Python Docker image that contains 2.6-3.3 and pypy. User could reference which version to use:

script:
  - pip2.7 install -r requirements.txt
  - python2.7 setup.py test
  - pip3.3 install -r requirements.txt
  - python3.3 setup.py test

I'm pretty sure this isn't the best way. You mentioned sub builds and that got me thinking about:

image: base-image
env:
  - DEBUG=true
builds:
    build:
        image: python
        env:
            - SECRET_KEY=123
        script:
          - pip install -r requirements.txt
          - python setup.py test
        services:
          - redis
notify:
  email:
    recipients:
      - brad@drone.io
      - burke@drone.io

But building matrixes, like on Travis, with this will possibly end up with a massive .drone.yml. Travis makes that pretty cool, I just set the versions of the language and additional environment variables. Every environment variable items in the env array trigger individual builds. I think we might be able to find a way to do that with Drone's philosophy.

image:
    name: python
    config:
        versions:
            - 2.7
            - 3.3
env:
  - DEBUG=true SECRET_KEY=123
script:
  - pip install -r requirements.txt
  - python setup.py test
services:
  - redis
        config:
            versions:
                - 2.6
                - 2.8
notify:
  email:
    recipients:
      - brad@drone.io
      - burke@drone.io

This example would trigger 4 sub builds one for each version of python with each version of the redis service. Hope this is somewhat useful.

[bradrydzewski]

I really like your suggestion. Instead of "versions" we could call them "tags" which is consistent with the docker terminology:

image:
    name: python
    tags:
      - 2.7
      - 3.3

Do you have any suggestions for a notation that would split a build into parallel tasks? For example, I only want to test against Python2.7, but my tests take a long time, so I want to break them up into suites and run in parallel.

We anticipated this change, so our database already supports sub-builds / matrix builds. The real challenge here is the yaml :)

[wilmoore]

Good discussion. Just a suggestion though: perhaps, rename the issue because this problem isn't exclusively a python concern. This issue is pertinent to other environments such as NodeJS, Ruby, Erlang/Elixir, etc.

Also, 👍 - with Travis (as much as I love it), trying to build https://github.com/exercism/exercism.io/blob/master/.travis.yml is difficult. You have to circumvent the magic with multiple bootstrap files or the equivalent.

By leaning on docker images (or dockerfiles), seems like single-language builds would be less magical in general and multi-language builds would be less obtuse.

[electrical]

Hi all.

Got directed to this thread by @bradrydzewski giving my £ 0.02
I think the travis matrix would be a good starting point to begin with. ( see https://github.com/elasticsearch/puppet-elasticsearch/blob/master/.travis.yml as example )

Most important parts i think are :

• enabling different versions of the selected language
• When having the same ENV variable, add it to the matrix ( In my case PUPPET_VERSION, but if you have an other one, add it to the matrix )
• Allowing failures for certain cases ( combo of language version and ENV variables )

For deployments its hard to choose when its allowed to do it ( when a certain test passed or all of them )

That's all i can think of at the moment.

[ewr]

For another example of a tool that is thinking along these lines, check out Test Kitchen's platforms and suites:

http://kitchen.ci/docs/getting-started/adding-platform
http://kitchen.ci/docs/getting-started/adding-suite

While Kitchen is really thinking in terms of OS versions, the issue here in terms of Rubies or Pythons is really the same thing a level up the stack.

[benallard]

I don't think the ability to paralellize one build is linked to this issue. As if you want to do that, you need to define (independant) sub-units of your build, which is pretty orthogonal to the idea of running the build multiple time on different environments ...

Try not to bloat this issue to much by adding every possible future feature to it. I think it's better to keep busy on one aspect at a time.

I think you came up pretty far there by defining to more aspect there: We need the ability to select different 'tags' of an image, and/or the ability to select different version of a service, and/or the ability to run the build with a different set of environments variables, and so on ...

I believe all of them could be implemented independently from each other ...

[bradrydzewski]

@benallard I definitely agree

I had a great discussion with an Ops lead that suggested adding a matrix section, where the axis could be defined. What does everything this of this proposal?

image: python:$$python_version
env:
  - DEBUG=true
  - SECRET_KEY=123
  - DJANGO=$$django_version
script:
  - pip install -r requirements.txt
  - python setup.py test
services:
  - redis:$$redis_version

matrix:
  python_version:
    - 2.7
    - 3.2
  redis_version:
    - 2.6
    - 2.8
  django_version:
    - 3.0
    - 4.0

this would end up producing 8 different sub builds. I think it is probably the most flexible design, but I'd love to hear what others think.

note that the matrix parameters should be handled in a similar manner to private environment variables. They can be injected directly in the script (using find / replace) using the $$ convention. They would also be injected directly into the build as environment variables.

[electrical]

Looks good to me @bradrydzewski :-) i would go for that.

[jpadilla]

@bradrydzewski that's pretty interesting right there. It took me a moment to figure out that the matrix defines the variables and its values, but I think it definitely covers all the cases we previously discussed.

[mdshw5]

Looks perfect to me. This is the only features keeping me from using drone right now!

On Feb 25, 2014, at 4:20 PM, Brad Rydzewski notifications@github.com wrote:

@benallard I definitely agree

I had a great discussion with an Ops lead that suggested adding a matrix section, where the axis could be defined. What does everything this of this proposal?

image: python:{{ python_version }}
env:

• DEBUG=true
• SECRET_KEY=123
• DJANGO={{ django_version }}
  script:
• pip install -r requirements.txt
• python setup.py test
  services:
• redis:{{ redis_version }}

matrix:
python_version:
- 2.7
- 3.2
redis_version:
- 2.6
- 2.8
django_version:
- 3.0
- 4.0
this would end up producing 8 different sub builds. I think it is probably the most flexible design, but I'd love to hear what others think.

—
Reply to this email directly or view it on GitHub.

[electrical]

An other important addition is to be able to tell which combo's are allowed to fail.
In my case im running multiple Puppet versions against different Ruby versions.
Some earlier puppet versions don't work against Ruby 2.0.0 and fail.

I was thinking of the following:

Allow all ruby 2.0.0 cases to fail:

allowed_fail:
    ruby:
      - 2.0.0

Allow Ruby 2.0.0 with Puppet 2.7.0 or 3.0.0 to fail

allow_fail:
  ruby_version:
    - 2.0.0
    puppet_version:
      - 2.7.0
      - 3.0.0

Any thoughts about it?

[benallard]

You should pay attention not mixing a notification issue with a fundamental architecture one ...

Do you don't want those test to run, or do you just don't care about their result ? If the former, this should be analysed there, if the later, we should figure out later about the right way to perform this.

Anyway, to extend on your idea, it should be possible to define sub-matrices where the build should not be performed.

I suggest the following syntax:

matrix:
  python_version:
    - 2.7
    - 3.2
  redis_version:
    - 2.6
    - 2.8
  django_version:
    - 3.0
    - 4.0
  except:
    -
      python_version: [2.7]
      django_version: [3.0, 4.0]
    -
      redis_version: [2.6]
      django_version: [4.0]

This would run all the builds except the 6 excluded ones ...

[gonzojive]

This was merged with #159, so I'm continuing discussion here. My use case is as follows:

• There are multiple projects in the same git repository. These projects have different build systems, languages, names, and teams. I'd prefer to keep all the projects within the same git repo because of many shared pieces of code.
• For each project, I'd like to see details about which tests are passing and failing, and at which point which tests began to fail.

It seems like the proposals so far don't solve the problem of having multiple projects per git repository. They do deal with the problem of multiple builds per project.

Personally I'm skeptical the ideal solution involves sticking purely with yaml files. As a user, I'd prefer if Drone stayed out of the way as much as possible and allowed me to script the configuration if I wanted:

for python_version in [2.7, 3.2]:
  for redis_version in [2.6, 2.8]:
    for django_version in [3.0, 4.0]:
      addBuild(
         variantName = "py=%f, redis=%f, django=%f" %
           (python_version, redis_version, django_version),
         image = "....")

Once again, this does not solve the multiple-projects-per-repository problem (maybe we should continue that discussion in #159), but hopefully it helps with the discussion at hand.

[mdshw5]

@gonzojive I don't think this works too well because then you are just picking a scripting language (looks like Python) which creates two problems:

1. You shouldn't use code to describe data when the actual data (a markup file) will do fine.
2. Non-Python users will be left wondering why Python was chosen over their favorite language.

As a side note, if this were a viable solution then something other than nested for-loops would be more readable:

python_version = (2.7, 3.2)
redis_version = (2.6, 2.8)
django_version = (3.0, 4.0)
for python, redis, django in itertools.product(python_version, redis_version, django_version):
  add_build(python, redis, django)

[gonzojive]

I think you can stick with a configuration language, but it'd be good to allow scripting if desired. You can do it in a way that leaves the choice of programming language up to the user.

Perhaps the .drone.yml file can include a line like

drone-config-script:
- ruby gen-build-plan.rb > $OUT

[bradrydzewski]

@gonzojive this is definitely a more advanced use case you are proposing. This project is still very young (0.1 alpha release) and the immediate focus is on the more simple use cases that serve 80+% of users. I'm happy to revisit this request in a few months once the project is further along.

[justone]

How about the use case of builds with separate deployments? For instance, it might be convenient to keep a project's source and website in the same repo. Then, when the build happens, the website is built and deployed to its server and the source is compiled and that result is uploaded to s3.

[fudanchii]

@justone If both of the source and the website sit on the same branch you can use deploy and publish plugin together in a single .drone.yml file.

[justone]

@fudanchii Interesting idea. I didn't realize you could have a deploy and a publish in the same file. I wonder if it's possible to have multiple deploys of different types in the same .drone.yml file. Like a git and an ssh deploy, each going to different places. I suppose that if #201 is merged, you can have one bash deploy that sends application artifacts to one location and the generated website to another.

If multiple builds can be specified, I think it would be good for there to be an environment variable injected into the build so that any deploy or publish can know which one it's working on.

[Linuturk]

Wanted to add my +1 to this. I'd like to have a situation where I can define multiple images (Ubuntu versions) to test my software. Something like this makes sense to me:

image:
  - ubuntu:14.04
  - ubuntu:12.04

[bradrydzewski]

@justone yes you can have multiple deployment entries in the yaml (ie ssh and git). We loop through each entry and execute.

[drewvanstone]

@bradrydzewski Adding my +1 to this. I'd also like to see this support parallelization. I think it would be a subsection to 'script', where you define which container to run a test in. I've modified your example above to illustrate it:

image: python:{{ python_version }}
env:
  - DEBUG=true
  - SECRET_KEY=123
  - DJANGO={{ django_version }}
script:
  container1:
    - pip install -r requirements.txt
    - python setup.py test
  container2:
    - pip install -r requirements.txt
    - python setup.py test
services:
  - redis:{{ redis_version }}
matrix:
  python_version:
    - 2.7
    - 3.2
  redis_version:
    - 2.6
    - 2.8
  django_version:
    - 3.0
    - 4.0

Love to hear other's thoughts on this too.

[Linuturk]

I was speaking with someone yesterday about this, and he suggested we might approach this with multiple YAML docs in a single .drone.yml

Something like this:

---
image: mischief/docker-golang
env:
  - GOPATH=/var/cache/drone
script:
  - go build
  - go test -v
services:
  - redis
notify:
  email:
    recipients:
      - brad@drone.io
      - burke@drone.io
---
image: mischief/docker-golang
env:
  - GOPATH=/var/cache/drone
script:
  - go build
  - go test -v
services:
  - redis
notify:
  email:
    recipients:
      - brad@drone.io
      - burke@drone.io

Obviously, all the options could be different between the two docs, and it would probably be easier to implement a second build using the existing code rather than restructure into a matrix style.

[drewvanstone]

@Linuturk I like that duplicating gives you more flexibility, but I feel 90% of use cases would just be duplicate configuration. For instance, if I wanted to run the build in 5 containers, I now have 5 portions of the YAML file where only the script section changes.

[bradrydzewski]

@davidak there are no restrictions as to how or where matrix parameters get injected into the yaml file. You could test multiple distros by doing something like this:

image: $$docker_image
[...]
matrix:
  docker_image:
    - ubuntu:14.04
    - ubunut:12.04
    - centos:latest
    - debian:wheezy

The ability to inject parameters with $$ is quite flexible

[bradrydzewski]

@os12 you will be able to invoke distinct scripts using the following notation:

build:
  image: go
  commands:
     - ./scripts/$$script
matrix:
    script: 
      - ci-debug
      - ci-opt
      - ci-opt-asan

and you could inject environment variables like this:

build:
  image: go
  commands:
     - export $$env
matrix:
    env: 
      - VARIANT=debug
      - VARIANT=opt
      - VARIANT=opt INSTRUMENTATION=asan

I'm pretty confident that all use cases are supported with the proposed yaml format and injection strategy proposed in #6 (comment).

[os12]

Both look good. Thanks Brad!

[bradrydzewski]

quick screencast of Drone running matrix builds from the command line: https://twitter.com/droneio/status/577496429242912769

[jpadilla]

@bradrydzewski that's great to see, great work!

[pksunkara]

Any update on when this is going to land?

[bradrydzewski]

yes, I expect to have a working 0.4 distribution available around May 22nd (sourced from the bolt branch). It will then take between 2 and 4 weeks before it is merged into master, depending on how quickly we are able to test, fix bugs, backport existing plugins to the new plugin model, document changes, etc.

[stevenpall]

@bradrydzewski Will this pull request also allow for dependant builds (i.e. project b should be built after project a)? I think this is a bit different than a matrix build if I understand correctly.

[bradrydzewski]

@stevenpall we are launching a new plugin model (included with the matrix feature) that would allow you to augment builds. I think this dependant builds could probably be added as a plugin.

[jakirkham]

+1 Refreshing approach to the build matrix. It's nice to see that they can be include in the configuration file and injected into the build environment.

[romani]

@bradrydzewski, any plans (dates) for release of this feature ?

[bradrydzewski]

@romani the 0.4 branch includes matrix builds, but it is still a bit too unstable and undocumented for people to start using. The 0.4 release, however, is still planned for this month

[donatj]

👍

[maci0]

+1

[bradrydzewski]

just wanted to give a quick update here.... I gave an implementation overview and live demo in last week's Rancher meetup of upcoming matrix build functionality. Link to the youtube recording:
https://youtu.be/86u8pVESbPQ?t=4586

[Ablu]

WOW. This looks pretty cool (not only the matrix build, everything). Can't wait for the release :)

[mjschultz]

@bradrydzewski I haven't had the chance to watch the video but is there a doc specifying a build env/commands to build and try 0.4? Once I get that far I'll be able to help as much as possible but figuring it out on my own is too much time for me.

[rdeutz]

@mjschultz as far as I know the only documentation is in this issue

[bradrydzewski]

Merged into master. Note that this is the initial implementation of matrix builds. Now everyone can start following the open issue for parallal matrix builds (#1254) targeted for the 0.4.1 release 😄

You can find the matrix documentation here:
http://readme.drone.io/usage/matrix/

Enjoy!