METRON-705: Parallelize the build in travis to the extent that is obvious

[cestella]

Travis suggests here that for situations where the integration get chunky, one can parallelize them using their build matrix functionality. Also, if we can separate those out, we can also process-parallelize the unit and build.

Currently the build time is cut roughly in half to 24 minutes wall-clock.

NOTE: This is just a stopgap that requires no code changes to lower build wall-clock times. This is not intended to replace work parallelizing the integration tests or making the build take less time.

[ottobackwards]

So to test this - if you have travis set up for your git account, I think you just push it to a branch in your repo and let travis build it?

[mmiklavc]

@ottobackwards Yes, that should do it.

[cestella]

Yep, that's how I did it. You can see a build of it on my account here

[ottobackwards]

I'm doing it now

[ottobackwards]

https://travis-ci.org/ottobackwards/incubator-metron/builds/199370356

#14 passed

Elapsed time 43 min 10 sec
Total time 1 hr 6 min 21 sec

[ottobackwards]

integration tests took 43 minutes

[cestella]

Wow, that's interesting @ottobackwards ; seems like this is very variable. Could you run it again and see if it's consistent?

[cestella]

Looking at those logs, it appears that the first phase, the build (not the actual integration tests) is 4 minutes 39 seconds in my integration-test phase build on travis vs 21 minutes 53 seconds on yours

[ottobackwards]

you bet.... btw -the build on this pr isn't even going at all

[cestella]

@ottobackwards Yeah, we have to use the apache job queue (which has 30 parallel builds) on travis rather than the big one for all open source projects. I suspect it gets slammed during the day.

[cestella]

Ok, this was just too volatile. I also got 43 minutes for the integration test run. It's probably worth investigating in the future, but for now I'm going to revert to just one container:

• parallelize the package phase
• parallelize the unit test phase
• run the integration tests in serial

[cestella]

Also, with only 30 parallel builds available across all apache projects, asking for 2 separate containers per build might be greedy. ;)

[ottobackwards]

#14.1 passed

Elapsed time 23 min 28 sec
On my second try around 5pm

[cestella]

I reran this 7 times on my travis account for the most recent commit and the times were on average 32 minutes 53 seconds

• 28m 17s
• 28m 13s
• 46m 28s
• 35m 53s
• 26m 26s
• 33m 0s
• 31m 57s

[justinleet]

+1, I was able to run this up in Travis setup on my repo without issue and it ran right in the middle of the time range listed