Add interleaved benchmark execution for test-suite

[aemerson]

This adds support for separating build and test phases, and running
benchmarks from multiple builds in an interleaved fashion to control
for environmental factors (ambient temperature, general system load etc).

New options:

• --build-only: Build tests without running them
• --test-prebuilt: Run tests from pre-built directory
• --build-dir: Specify build directory (used with --test-prebuilt)
• --exec-interleaved-builds: Comma-separated list of builds to interleave

Usage:

1. Build two compiler versions:
   lnt runtest test-suite --build-only
   --sandbox /tmp/sandbox-a
   --cc /path/to/clang-a
   --test-suite ~/llvm-test-suite
   ...

   lnt runtest test-suite --build-only
   --sandbox /tmp/sandbox-b
   --cc /path/to/clang-b
   --test-suite ~/llvm-test-suite
   ...

2. Run with interleaved execution:
   lnt runtest test-suite
   --sandbox /tmp/results
   --exec-interleaved-builds /tmp/sandbox-a/build,/tmp/sandbox-b/build
   --exec-multisample 3

   This runs tests in the pattern:

   • Sample 0: build-a -> build-b
   • Sample 1: build-a -> build-b
   • Sample 2: build-a -> build-b

Or, test single build:
lnt runtest test-suite --test-prebuilt
--build-dir /tmp/sandbox-a/build
--exec-multisample 5

Reports are written to each build directory (report.json, test-results.xunit.xml, test-results.csv).

[aemerson]

• Generalize --build-dir to work with all build modes #108
• Add interleaved benchmark execution for test-suite #107 👈 (View in Graphite)
• main

This stack of pull requests is managed by Graphite. Learn more about stacking.

[ldionne]

I'm not super familiar with the "run the test suite" part of LNT, so this might not be the most sensible question. But have you considered making this into a new subcommand of lnt runtest instead of adding "a new mode" for lnt runtest test-suite? This seems to be doing things quite a bit differently from the regular lnt runtest test-suite, and I (perhaps naively) wonder if it wouldn't be better to acknowledge that and not try to make it fit into lnt runtest test-suite.

Like I said, I don't have a strong opinion, but I wanted to at least ask the question.

[aemerson]

I'm not super familiar with the "run the test suite" part of LNT, so this might not be the most sensible question. But have you considered making this into a new subcommand of lnt runtest instead of adding "a new mode" for lnt runtest test-suite? This seems to be doing things quite a bit differently from the regular lnt runtest test-suite, and I (perhaps naively) wonder if it wouldn't be better to acknowledge that and not try to make it fit into lnt runtest test-suite.

Like I said, I don't have a strong opinion, but I wanted to at least ask the question.

Good question. I'm no expert on this project, but from what I can tell the test runners seem to handle most of the actual mechanics of building and testing (and for example, implementing PGO flows). So moving this mechanism into the runtest level of abstraction doesn't seem possible.

I don't know of anyone actively using the other test runners, but if there is then maybe we can think about re-working the whole thing to allow the runtest handler to be aware of such concepts like "build directory" and "execution phase".

I've addressed your other comments now.