🏗 Matrixify Experiment Build/Tests on CircleCI

[danielrozenberg]

This PR does DRY

Description from previous version of this PR that included splitting integration and e2e tests into separate jobs:

Reduces total running time of CircleCI for PRs from ~20.5 minutes (example) to ~18.5 minutes (example), at the cost of about an extra ~8.5 minutes of total VM time. Not sure if it's worth it, wdyt?

[amp-owners-bot]

Hey @rsimha! These files were changed:

.circleci/config.yml

[rsimha]

Nice exploration! Unfortunately the additional cost probably makes this not worth it.

A few reasons:

• CircleCI gives our entire org a max of 80 parallel VMs at any given time. If demand exceeds that, queueing will ensue.
• Each extra VM we use incurs a $$ cost for CPU minutes spent during start up (~2 mins per new VM, as things stand today)
• In just the past 24h, our peak parallelization has risen from 9 to 11, and this PR will make it 14. That means fewer people will be able to run builds at a time.

Alternative challenge: If we can bring down the running time of package installs from ~1 min to a few seconds (I'm working on this), gulp dist from ~8 mins to ~2 mins (@samouri is doing this), and gulp e2e from ~10 mins to ~5 mins (needs a volunteer), we can get down to sub ~15 minute builds without any increase in VM consumption.

[danielrozenberg]

ok! So I'll revert the 2nd commit for this PR and keep it around in case we improve the overhead enough to make it worth it

[danielrozenberg]

Done, PTAL

[rsimha]

👏👏👏 LGTM! (I'm kind of disappointed in myself that I didn't do it this way to begin with.)

[danielrozenberg]

LGTM! (I'm kind of disappointed in myself that I didn't do it this way to begin with.)

Hey, you switched an entire CI system... figuring out the subtleties comes later ;)

[samouri]

TIL You could do this!