🏗️ enable 3-at-a-time batching for all sauce labs tests

[rcebulko]

Comparing with #27016 to see batch size impact on Travis running time

[amp-owners-bot]

Hey @estherkim, these files were changed:

build-system/pr-check/remote-tests.js
build-system/tasks/runtime-test/helpers.js

[rcebulko]

For context/discussion/analysis of the ideal batch size, see #27016

[rsimha]

Thanks for the exploration and for putting together this PR. I'm adding a blocking review because this PR as written will break CI builds. Suggestions to fix are in the comments below.

[rcebulko]

Not to be addressed at this moment, but something I noticed when looking at the batching code. It doesn't continuously run "up to $BATCHSIZE batches". That is, when it runs 6 tests with a batch size of 3, it waits until all 3 complete before triggering another 3 at once. Instead of running in batches like this, we could probably speed up runtime by using a pool instead. This way, as soon as one browser finishes, another can start.

[rcebulko]

It appears we also re-transform tests with browserify once each batch; I'm guessing we should be able to do that once for all the browsers, stable and beta, instead of doing it per-batch. Each browserify invocation takes a minute or so.

[estherkim]

Thanks! Do you know why we're testing on Chrome twice? https://travis-ci.org/ampproject/amphtml/jobs/657948850#L2400-L2402

Also, can you update the comments here

amphtml/build-system/tasks/runtime-test/helpers.js

Lines 250 to 252 in 46434f4

	* If --stable is provided, runs tests only on stable browsers in sauce labs without batching.
	* If --beta is provided, runs tests only on beta browsers in sauce labs without batching. Does not fail.
	* If neither --stable nor --beta are provided, runs test on all browsers in sauce labs with batching.

Withdrawing my blocking review now that test status reporting is no longer broken. Will defer to @estherkim for final approval of the code changes. LGTM from my side.

[estherkim]

Can we flip this so that stable browsers are set when argv.stable is set? And if neither flags are present, default to all or stable, up to you.

[rcebulko]

Done, PTAL

[estherkim]

Thanks!