Parallel package builds

[dennybaa]

Current build process pipeline is in bash, and it's nightmare to support it. Introducing parallel package build will bring more bash hardcore code. We should waste the current build controlling code in favor of a better solution.

To leverage parallel package build (need by #7) the best option is to use flow based build tool with support for dependencies and parallelism. The current vision is to use rake + sshkit.
Rake is a prefect candidate since it's provides a very rich set of features and it has a very simple DSL.

---

Other python based solutions will take more effort to use. There are following options:

• For build tools:
  • http://pydoit.org/
  • http://www.celeryproject.org/
  • https://github.com/seomoz/shovel
• For remote ssh control:
  • This library seems to be a very useful and really cool stuff https://plumbum.readthedocs.org/en/latest/

Unluckily all of the python based tools either lack needed parallelism or just quite heavy to use...

[dennybaa]

Rake + sshkit work nice. I've managed to extend sshkit behavior to capture output of concurrent commands. This was a bit hard, since it has required to implement threaded dispatcher which consumes output of concurrent processes, mangles it and finally outputs in a nice and readable way.

Talking about batch tools and parallel ssh command executors, I can say none of them supports parallel execution on a "single host". Neither sshkit nor fabric.

So what I come with is rake with a build-in enhanced sshkit which allows a user easily define parallel jobs and get all the power of ssh connections via sshkit. Here's an example DSL:

require 'rake'
require './rake/remote'

ssh_options = {
   keys: %w(/root/.ssh/busybee),
   auth_methods: %w(publickey)
}


buildnode = ENV['BUILDNODE']
# testnodes = ENV['TESTNODES']

Remote.output_verbosity = :debug
remote = Remote.new(ssh_options)

# =============================================================================

# Run shellout dispatcher thread
ShellOut.run

# Flushes shellout when tasks are finished, finalization is required
# to write out any messages left in the queue.
multitask :build => [:first, :second] { ShellOut.finalize }
task default: :build

task :first do
  remote.ssh hostname: buildnode do
    options label: 'build st2common'
    execute :bash, '-c', %q("ls -l /bin | head -n15")
  end
end

task :second do
  remote.ssh hostname: buildnode do
    options label: 'build st2auth'
    execute :bash, '-c', %q("ls -l /bin | head -n15") 
    execute :bash, '-c', %q("echo fail it; /bin/false"), label: 'fail it'
    execute :true
  end
end

This kind of DSL makes is easy to understand and far much better then any killing bash stuff, just have a look at the picture of parallel tasks example:

[arm4b]

^^ 👍 Really nice output!

[dennybaa]

Okay this was merged into master. Build for wheezy takes 9 min on a 2CPU instance!

[arm4b]

Cool thing to have 👍

Probably candidate for closing or you plan to extend it like build 1 package = 1 node approach for future horizontal scaling?

[dennybaa]

Okay, we've reached the same performance on CircleCI. So far I'm satisfied with the current results, further improvements can be delayed for now.

[dennybaa]

implemented

[estee-tew]

https://stackstorm.slack.com/archives/stackstorm/p1454357997003953