Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We鈥檒l occasionally send you account related emails.

Already on GitHub? Sign in to your account

Maintenance: Sharding and parallelizing RSpec tests in CI #2801

Merged
merged 41 commits into from Apr 6, 2020

Conversation

kevinrobinson
Copy link
Contributor

@kevinrobinson kevinrobinson commented Apr 3, 2020

This started as experimental, but going to ship it now 馃殺

This PR cuts total build time to ~4min! The only downside was that the way the CoverageChecker worked each shard would fail, because none of the partial test runs reach 100% coverage on the expected files. So I updated this to upload the coverage artifacts, and then added another step to merge them and run the coverage check.

This ran into a few issues:

  • SimpleCov's collate function seemed to work but the methods that it normally provides had missing data (eg, filenames). So this still runs SimpleCov on each test run to produce artifacts, but uses the plain Ruby Coverage output it produces for merging the shards on its own in plain Ruby. This also lets the coverage enforcer step us just plain Ruby rather than all the gems etc.
  • The GitHub download action didn't support multiple files, but the v2-preview does. That has a bug with not properly unzipping artifacts though, so we work around that in here, with comments to the upstream issue which should be resolved soon.
  • Unrelated: there were some severe latency spikes with cached folders for Ruby gems. I commented upstream and it hasn't come up again this afternoon.
  • Unrelated: there's an incident now impacting node download servers, so this works around and adds comments on that.
  • Unrelated: Tuned down the Postgres health check, it was waiting ~10 seconds longer than needed.

EDIT2: Verifying code coverage check works, link:
Screen Shot 2020-04-06 at 3 55 31 PM

EDIT3: example:
Screen Shot 2020-04-06 at 4 02 05 PM

EDIT4: 馃帀
Screen Shot 2020-04-06 at 4 33 00 PM

@kevinrobinson
Copy link
Contributor Author

kevinrobinson commented Apr 3, 2020

This works great, and would cut total build time to ~4min! Seems worth doing because of that; the only downside is that the way the CoverageChecker works now each shard fails, because none of the partial test runs reach 100% coverage on the expected files. We could update that or tackle it another way, and the 2x speedup cutting off over 5min on the build time seems worth it, so will consider.

@kevinrobinson kevinrobinson changed the title Maintenance: Experiment with sharing and parallelizing RSpec tests in CI Maintenance: Experiment with sharding and parallelizing RSpec tests in CI Apr 3, 2020
@kevinrobinson kevinrobinson changed the title Maintenance: Experiment with sharding and parallelizing RSpec tests in CI Maintenance: Sharding and parallelizing RSpec tests in CI Apr 6, 2020
@kevinrobinson
Copy link
Contributor Author

kevinrobinson commented Apr 6, 2020

selfie

@kevinrobinson kevinrobinson merged commit e2f8ed3 into master Apr 6, 2020
10 checks passed
@kevinrobinson kevinrobinson deleted the patch/parallelize-rspec branch Apr 6, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

1 participant