[RunAllTests] Fix #3752: Introduce Bazel test batching in CI

[BenHenning]

Explanation

Fixes #3752

This PR introduces support for batching Bazel tests in CI, that is, rather than running all tests under a single workflow (no sharding) or running one workflow for each test (maximum sharding), the system will now compute "buckets" of tests to run together in order to:

• Avoid the max shard limit of 256 instituted by GitHub
• Benefit more from Bazel build & runtime sharing by reducing non-Bazel Actions overhead

Recent runs suggest a substantial reduction in runtime: 2-2.5 hours down to ~45 minutes (a bit slower than Gradle on the last attempt, though this will be greatly affected by overall CI load in the organization, how much the local PR changes the build graph, and overall performance of the cache since it's eagerly evicted). A subsequent run completed in ~30 minutes (10-15 minutes faster than Gradle) after bucketing tweaks (and probably with a warmer cache). The overall runtime is still substantially longer than Gradle (see table below), but Bazel lends itself to parallelization much more flexibly than Gradle does. Edit: after introducing improved batching policies & increasing run rate to 10 parallel rules, we actually saw an ~18 minute runtime which is actually faster than even the non-app Gradle test suite.

Some notes on the batching strategy:

• There are two aspects to decide when organizing buckets: what should go into each bucket, and how big should each bucket be
• The first decision (referred to as 'grouping') is done simply by first organizing tests based on their top-level package (e.g. 'app', 'domain', 'scripts', etc.), then grouping them based on observed performance (app, domain, and scripts are all run in their corresponding groups, and all other tests are combined under a single 'generic' bucket to improve overall throughput)
• The second decision (referred to as 'partitioning') was decided by observing actual runtime of the tests. In general, we try to batch a lot of tests together (50), but scripts were limited to 25 and app tests to 15 due to observed runtimes (see table below)
• This batching strategy is meant to optimize the worst case scenarios (i.e. all tests running), but it's expected that lesser-than-all cases (where smaller test runs should become more common as modularization continues) will still run well but perhaps not as relatively optimally as the worst case

The result is a 20x reduction in total shards (down to 12 for the whole test suite) and shards that each require about 15-20 minutes to run. We likely will continue to tweak how these are balanced over time, and long-term could even consider making these parameters automatically tuned based on long histories of test runs.

The following table shows the first fully successful test run (pre-grouping) which was used to influence grouping & partitioning policies:

Test bucket	Cache group policy	Test count	Observed shard count*	Max tests/shard policy	Total runtime	Average runtime
app	app	117	6	15	99 min 7 sec	~50.83 sec/test
data	generic	13	1	50	6 min 55 sec	~32 sec/test
domain	domain	73	4	50	34 min 34 sec	~28.4 sec/test
instrumentation**	generic	4	1	50	5 min 41 sec	~85.25 sec/test
scripts	scripts	23	2	25	19 min 3 sec	~49.7 sec/test
testing	generic	12	1	50	3 min 43 sec	~18.6 sec/test
utility	generic	16	1	50	6 min 12 sec	~23.25 sec/test

* This corresponds to the number of shards observed when running with a non-grouping 20-tests-per-shard policy.

** Note that results for instrumentation tests are probably unreliable due to the low number of tests, so it's currently being considered a 'fast' test bucket.

Other technical details of note for this PR:

• This PR enables caching for scripts CI runs (except third party licenses) to improve their performance
• Since the main way to pass information from the bazel_compute_affected_targets workflow to the shard workflows is through an actions matrix, this PR encodes that information in a Base64 representation of a GZIP-compressed proto binary. While this has some nice benefits, it unfortunately leads to unreadable workflow names (see Checks tab). This is being partly mitigated by including the test bucket & shard number as a prefix, though users really just need to look at the workflow output to see which tests are managed by that particular shard.
• The [RunAllTests] mechanism has been simplified by consolidating its behavior into a new boolean parameter passed to the Kotlin script, reducing the workflow complexity a bit
• The run test workflow now builds the tests before trying to run them. This is by design so that they can be built in a while loop to work around Fix occasional Jetifier build failures in Bazel #3759 (note that I've still only observed these locally, but for the sake of limited development time at the moment I didn't investigate if they occur on CI, especially since the existing caching mechanism makes it harder to reproduce these situations without forcing caching off & waiting a really long time for builds)
• ComputeAffectedTestsTest can now exceed 300s of runtime, so it's been updated to have Bazel-based sharding (for faster local runs) and to indicate that its size is large so that Bazel doesn't kill it if it exceeds 300s total runtime
• The bazel_compute_affected_targets workflow now uses caching a bit weirdly: it pulls the scripts cache for a fast base64 decode of the proto, then another fetch for the build cache it needs for its tests. Fortunately, the latter is the one that's actually updated. actions/cache doesn't currently support disabling cache uploads (which is what we'd want in this case for the script).
• Javalite needed to be updated to work correctly in the scripts environment since we were way out-of-date; this seems to be working fine with the latest IntelliJ Bazel plugin, AS Arctic Fox, and Bazel 4.0.0; I didn't bother updating Gradle here. Interestingly, this actually required us to change our dependency ('lite' to 'javalite'); there must have been a dependency rename in the past couple of years since we've last updated.
• The computed targets intentionally inject randomness when computing shards to try and benefit from an amortized cache rather than arbitrary environmental factors leading to unexpected biases toward certain test lists or buckets (it's essentially a "fair" policy)

Finally, note that while this PR demonstrates the '[RunAllTests]' flow, #3750 demonstrates the standard compute affected tests (though because it's changing a WORKSPACE file it's triggering the full set of tests to run).

Essential Checklist

• The PR title and explanation each start with "Fix #bugnum: " (If this PR fixes part of an issue, prefix the title with "Fix part of #bugnum: ...".)
• Any changes to scripts/assets files have their rationale included in the PR explanation.
• The PR follows the style guide.
• The PR does not contain any unnecessary code changes from Android Studio (reference).
• The PR is made from a branch that's not called "develop" and is up-to-date with "develop".
• The PR is assigned to the appropriate reviewers (reference).

For UI-specific PRs only

N/A

[BenHenning]

Initial failure state: https://github.com/oppia/oppia-android/actions/runs/1191450690.

[BenHenning]

This mid-sharding strategy is demonstrating excellent throughput; the Bazel tests almost ran as fast as the app module Gradle tests despite being more tests overall & generally slower when compared with Gradle. I'm going to see if we can calibrate the shards with some slight heuristics to get even better throughput.

[BenHenning]

NB: latest runtime for this case was 30 minutes for the full Bazel test suite (which suggests a lot of caching going on). That actually beats Gradle outright by a pretty large margin (usually app tests on Gradle take 40+ minutes).

[BenHenning]

Hmm. I actually think that this might a viable point to mark Bazel tests as required now. Let's see how this PR lands over the next week for practical usage, but if it behaves as well as we're seeing here then we probably don't even need to bother waiting for modularization to mark these workflows as required.

[BenHenning]

I'm fairly confident in this latest run passing, so sending into review now & merging downstream to unblock #3750.

[BenHenning]

@vinitamurthi @FareesHussain @anandwana001 this PR doesn't actually require any codeowner approvals, but I'd like your thoughts on the approach & implementation. PTAL.

[BenHenning]

Another thought: given that this reduces the shard load by 20x on GitHub, I think we could also consider changing the runtime rate to 10 max concurrent instead of 5. We're less likely to exceed our quota with fewer overall shards, and even if we hit it when a lot of concurrent PRs are being developed the cost to wait is way lower since each workflow comprises ~8% of the overall test runtime vs. ~0.4%. That will also result in the overall test suite completing in closer to 15-20 minutes which puts us back to pre-slow-down-Gradle days in early 2020 in terms of CI performance.

[BenHenning]

Latest Bazel run (with 10 max concurrent runs): 18 minutes.

[FareesHussain]

I'm traveling rn, I'll review this tomorrow

[seanlip]

@vinitamurthi @anandwana001 @FareesHussain friendly ping.

[oppiabot]

Unassigning @vinitamurthi since they have already approved the PR.

[anandwana001]

LGTM

[seanlip]

@FareesHussain is away, so let's merge this.