Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We鈥檒l occasionally send you account related emails.

Already on GitHub? Sign in to your account

Parallelize macOS tests #19446

Merged
merged 5 commits into from Jun 6, 2019

Conversation

Projects
None yet
2 participants
@as-cii
Copy link
Member

commented Jun 4, 2019

Refs: #19366

This pull request changes our CI configuration to run tests in parallel on macOS. Specifically, it splits the macOS step into a build phase and a test phase.

Build Phase

During this phase, Azure Pipelines takes care of bootstrapping and building Atom. It then uploads the produced artifacts so that they can be used by the test phase described below.

Note that, perhaps counterintuitively, we will still execute main process tests as part of this phase. This is unfortunate, but there seems to be an issue when trying to execute main process tests on a container different from the one Atom was built on. These tests usually take less than 10 seconds anyways, so they should have a negligible cost on parallelism.

Test Phase

During this phase, Azure Pipelines takes care of downloading the artifacts published by the build phase and use them to run tests. In particular, this phase is further split into 3 jobs that are executed in parallel:

  • macOS test core, which runs core renderer tests.
  • macOS test packages-1, which runs tests for the first 22 packages
  • macOS test packages-2, which runs tests for the remaining packages

Splitting tests evenly across containers is non-trivial, especially since we don't know how long each test suite takes to execute. We could probably download that information from previous builds and schedule tests accordingly, but in this pull request I decided to go for a more low-tech approach. Empirically, I observed that the splitting strategy illustrated above achieves a good level of parallelism without introducing a ton of complexity.

Conclusion

With these changes, build times when the node_modules cache is warm drop from ~1hr to ~40 minutes. In other words, after this pull request lands, we should be able to observe a consistent ~33% improvement in our build times. 馃悗

Please, note that there is still room for improvement, but it seems like additional enhancements to CI would come at a higher cost and we may want to work on them in a separate moment.

@as-cii as-cii force-pushed the as/ci-parallelize-macos-tests branch 2 times, most recently from 6f51ff7 to 0fdb013 Jun 4, 2019

@as-cii as-cii force-pushed the as/ci-parallelize-macos-tests branch from 0fdb013 to 791a583 Jun 5, 2019

@as-cii as-cii force-pushed the as/ci-parallelize-macos-tests branch from 5f7ae4f to 2b0e5b0 Jun 5, 2019

Run main process tests during build step
For some reason, one main process test fails when run in a different 
container than the one in which Atom was built. These tests are pretty 
quick, so their impact on parallelism should be negligible.

@as-cii as-cii marked this pull request as ready for review Jun 5, 2019

@as-cii as-cii requested review from rafeca and nathansobo Jun 5, 2019

@nathansobo
Copy link
Contributor

left a comment

鈿★笍 Put that in your pipeline and smoke it!

@as-cii as-cii merged commit 05d6752 into master Jun 6, 2019

0 of 2 checks passed

Atom Pull Requests #20190605.11 failed
Details
continuous-integration/appveyor/pr AppVeyor build failed
Details

@as-cii as-cii deleted the as/ci-parallelize-macos-tests branch Jun 6, 2019

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can鈥檛 perform that action at this time.