Test environment report

[roberth]

Is your feature request related to a problem? Please describe.

Nix runs in many environments, each of which needs testing.
Not all tests run in all environments.
Any non-trivial logic in test code can contain bugs, which will go unnoticed. Test conditions that control whether a test runs in particular are fail unsafe. (E.g. if system == "x86-64_linux" .... Did you spot it?)

Given the source of any test, I want to be able to tell reliably,

• Whether it runs at all. If not, CI should fail.
• In which environments/situations it is run.

Describe the solution you'd like

• Make all tests report whether they've run (esp. in case of the Nix package), and in which situations they've run in case of a test matrix.

• Write a derivation that aggregates all the test reports from all our builds (packages, checks, hydraJobs, etc)

  • Make it so that certain "feasible" subsets can be reported, e.g. reasonably quick tests on x86_64-linux with kvm.

• Make all derivations report which tests exists. Join this to the test run reports. If not a single test run report is found, fail.

  • The logic for finding tests that exist should be dead simple, such as grepping the source.
  • Also check the converse. If a test is run but not found by the simple logic, it has a bug.

Describe alternatives you've considered

Just hope.

Use an existing solution? Didn't find one. Most of our test code is custom, so I have doubts, but if anyone knows anything, please comment.
JUnit's report format is a standard that's exported by a number of frameworks. Would be nice to reuse that perhaps.
Maybe combine this with a cross referencing solution? Something like GHC Notes, or something that formalizes "grep across files with identifiers".
Combine with coverage report? Combine the coverage reports' per-line info? (Where we have coverage reporting; something else elsewhere)

Additional context

Priorities

Add 👍 to issues you find important.