Code coverage as a separate stage in CI

[connorshea]

Issue

Right now, the CI build is set up such that code coverage is another stage after the "main" tests are run. However, code coverage is only run after both the first stage completes successfully and the test suite is re-run to allow for the coverage checker to get coverage data.

This means that the test suite takes a lot longer than it needs to:

• If one of the tests in the first stage takes 10 minutes longer than the others, the code coverage stage has to wait for that extra 10 minutes before starting.
• The code coverage job runs the test suite again on PHP 7.0, which means that – even after the first stage is finished – code coverage needs to run for 20+ minutes despite the fact that the test suite has already run and passed on 7.0.

Current CI setup:

Expected Behavior

Code coverage shouldn't require re-running the job.

Possible Fix

There are two main ways to go about this:

• Run the code coverage job as part of the first stage, in the PHP 7.0 job. This has the downside of code coverage being reported even if other jobs fail, but IMO that isn't a big problem.
• Run the PHP 7.0 job with code coverage enabled, but don't report the data to codecov in that job. Instead, pass the code coverage data to the code coverage job in the second stage and have that second job report the data to codecov. I'm not sure how exactly to pass data between jobs in Travis, but I'd imagine it's possible(?). The benefit of this is that it would only report code coverage to codecov when all the tests pass, though it's more complex to set up.

This change would save about 20 minutes on successful runs (on failed runs there will be no difference) of CI.

[connorshea]

@Dillon-Brown I was told to ping you :)

[Dillon-Brown]

@connorshea Hmm, not sure how effective this would actually be.

• "If one of the tests in the first stage takes 10 minutes longer than the others, the code coverage stage has to wait for that extra 10 minutes before starting."

This is true however with Travis we only get 1 job 5 builds so while the codecov build is waiting to run it the freed up build will be running on another PR.

I do agree we need to speed up the CI and this is indeed a change I want to make. I would propose that we run all the builds on all the PR's in parallel while also splitting up the tests further (1 build running acceptance, 1 running unit, 1 running codecoverage). We wouldn't be able to do this with the Free version of Travis though, which is why I would like to move to Jenkins :)

[connorshea]

It's 5 jobs and no build limit AFAIK (of course, the number of builds is limited by the number of jobs you can run simultaneously), you can have up to 5 running on the same build, or you can have 5 builds with 1 job each. The point is more that I'd like to have the tests for a given PR finish faster, though it would still have an impact on the overall CI system as one of those job slots is taken up by the code coverage job while it's running.

Personally I'm a big fan of GitLab CI, but I worked there so I'm biased ;) Unfortunately I haven't gotten a chance to try GitLab CI's GitHub integration, so I don't know if it's any good.

It may also be worth checking out Circle CI, though I think the free plan might be limited to 1000 minutes/month, which would be a deal breaker.

Another option may be to contact Travis and request that they raise the limit for you?

[Dillon-Brown]

Done here: #7802 (Should have done this ages ago tbh). I unfortunately couldn't find an easy way to send files between jobs without uploading somewhere else first so I just opted to run the code cov as part of the 7.0 job.

[connorshea]

@Dillon-Brown should this be marked as resolved in next release?

[Dillon-Brown]

@Dillon-Brown should this be marked as resolved in next release?

Yes, thanks!