Speed up Mac OS Travis builds.

[robertnishihara]

Recently a number of our Mac OS Travis builds have timed out due to exceeding the maximum job length (somewhere around 45 minutes). We should speed these tests up.

[robertnishihara]

One solution is to split the tests across more Travis builds. E.g., create wheels for 3.4 and 3.5 in one Travis build and create wheels for 3.6 and 3.7 in another Travis build.

[robertnishihara]

One thing we can do here is move some of the stress tests to jenkins. Currently they take about 10 minutes and this would help.

[danielsuo]

Yeah, this is pretty annoying; I don't think I've ever successfully passed Travis in my short career here. A bit of a drastic suggestion, but what I've done before is expand on the Travis build matrix.

1. We want multiple build matrices, not just one. I specified this these matrices in Python and generated a "flat" .travis.yml file. It's some overhead on maintainers, but only needs to be run occasionally (or could just be done manually as it is now).
   • tests: {Linux, Mac} x {2.7, 3.5) x {test groups}
   • wheels: {Linux, Mac} x {2.6, 2.6, 3.5, 3.6}
   • other: {lint, valgrind}
2. Make this easier by creating more .travis scripts (e.g., have a few test scripts that divide up the current slate of test scripts into a few logical categories; roll up the valgrind tests)

As I said, this might be drastic; one simple way to start is just to split out the MacOS wheel jobs.

[robertnishihara]

It would be straightforward to split out the MacOS wheel jobs into separate Travis runs, so maybe we should do that. I was concerned that splitting Travis builds into multiple builds would slow things down (e.g., if there are substantial delays between when one build finishes and when Travis starts the next build), but maybe that isn't an issue.

See related discussion in https://issues.apache.org/jira/browse/ARROW-1781.

[danielsuo]

Hm, just want to confirm we're talking about the same thing; when you say multiple builds, do you mean something like 123.1 (i.e., build 123, job 1) and 124.1 or do you mean 123.1 and 123.2? I was referring to the latter (i.e., split up into more jobs within a build).

It is true that Travis is a best-efforts service, but given that each build can take an hour or more as is, it'd be good to have tests pass vs. not. Also, Travis can take advantage of concurrency, though of course that's best-efforts too.

I'd say after splitting out MacOS wheels, the next thing to do would be to split up tests (as you suggested earlier) so the MacOS tests pass. Had a look at the Arrow issue you linked, but hadn't looked carefully at what was holding the MacOS testing up (as opposed to Linux).

[robertnishihara]

I may not have been using the correct terminology. When you said "split out MacOS wheels", I interpreted that to mean modifying .travis.yml as follows.

Currently we have a section that contains

    # Build MacOS wheels.
    - os: osx
      osx_image: xcode7
      env: MAC_WHEELS=1

That would be changed to something like

    # Build MacOS wheels.
    - os: osx
      osx_image: xcode7
      env: MAC_WHEELS=1, PYTHON_VERSION=2.7
    - os: osx
      osx_image: xcode7
      env: MAC_WHEELS=1, PYTHON_VERSION=3.4
    - os: osx
      osx_image: xcode7
      env: MAC_WHEELS=1, PYTHON_VERSION=3.5
    - os: osx
      osx_image: xcode7
      env: MAC_WHEELS=1, PYTHON_VERSION=3.6

so Travis would launch 4 containers (instead of 1), and each would build a single wheel.

Is that what you had in mind?

[danielsuo]

Yep, that's right! Just pushed a quick test on my fork. Will post with a pull request for you to examine when I have a chance to review the results myself.

It a decent bit of duplicated code in .travis.yml, sadly.

[robertnishihara]

We could do more here, but this is largely done now by @justinwyang.