Support a subset of the test suite on PRs

[alexcrichton]

Over at rust-lang/rust we're using Travis to great profit right now, but our CI setup is slightly different from many others I've seen at least. Right now we have a very large test suite with ~30 matrix entries each taking ~2 hrs to complete, quite a lot of work on each full build!

The rate at which we receive PRs far exceeds the build capacity that we have to actually run a full build on all pull requests. This means that we actually run a subset of the test suite for PRs (just as a sort of opportunistic gut check). When we merge a PR we run the full test suite (via a separate automated system).

Technically what's happening is:

• PRs run only one entry of the matrix that we have (we whitelist this)
• Commits to the auto branch run the full test suite

We then have separate infrastructure for pushing PRs to auto and managing merging that back to master.

So today there's not really "official" support for running a subset of the test suite on Travis, so we sort of hack around this in our configuration. Specifically we have a special variable which is then checked on PRs. If not set the build exits quickly to not take up build capacity. This means that most of our builds have a full matrix with all but one entry exiting in a few seconds. (note that the one entry there which ran took ~1hr).

Unfortunately though while this strategy works it has a number of drawbacks:

• We rely on builds "exiting quickly" to not overload our capacity. This means we have to be very careful to structure all work to happen after the ALLOW_PR=1 check, which we forget to do sometimes. If you take a look at one of our PR builds you'll notice that one OSX entry took ~20 minutes even though it ended up not doing any work. Basically while it's possible for us to manually make sure that all our "pre check" work exits quickly we inevitably make mistakes.

• We still end up stressing our build capacity for all these PR builds. Most builds exit quickly but the time it takes for a matrix entry to get scheduled, run for a bit, and then exit is on the order of a few minutes (especially for the entire matrix). It'd save quite a few resources on everyone's end I think to skip scheduling these builds entirely!

So overall, it'd be awesome if there were something like skip_pr: true in the yml configuration for each matrix entry. We'd throw that in most of our entries (except one) and then we wouldn't have to worry about these drawbacks!

[BanzaiMan]

Hello. Thank you for taking the time to open this issue.

We have received requests of this sort (configure different build matrices/stages based on the requests' certain properties), and have an internal ticket to track them. I've added this to the ticket.

[kennytm]

This feature is now supported via conditional jobs, by using the if entry inside matrix.include. For instance skip_pr may be written as

matrix:
  include:
    - env: COMPLEX_TEST=0
    - env: COMPLEX_TEST=1
      if: NOT type = pull_request   # skip this for PRs.

[BanzaiMan]

As @kennytm mentioned, I believe this can be closed. Do let us know if there are additional things to address.

Thanks.