No convenience method to collect test report files

[dmivankov]

Description of the problem / feature request:

bazel test SOME_TESTS_EXPRESSION does not print paths to test reports, they will be stored under $(bazel info bazel-testlogs)/PACKAGE_PATH../TEST_LABEL. Which could be inconvenient for report collection

• just collecting all reports may pick up reports from previous invocations
• in order to collect specific reports one needs to resolve test expression (can have wildcards) and then transform paths, probably doable with appropriate baze ?query flavor and some scripting

As a workaround one could clear bazel-testlogs but that's not concurrent-command-friendly and feels hacky in general. Another option is change output_base per command but that may trigger other issues.

Feature requests: what underlying problem are you trying to solve with this feature?

Run tests in CI in several packs and collect reports separately

• can collect some reports early
• can handle reports differently (post-process for example)

What operating system are you running Bazel on?

NixOS

What's the output of bazel info release?

release 4.2.1- (@non-git)

If bazel info release returns "development version" or "(@non-git)", tell us how you built Bazel.

NixOS nixpkgs

Have you found anything relevant by searching the web?

• request to depend on test results in rules Allow targets that depend on test results #11240 is related as it can unblock post-processing per test group (but maybe only in cases without wildcards)
• request to report build output paths Obtaining output paths of the build artifacts #8739
• another report post-processing request Allow providing an external test.xml producer #4507

Any other information, logs, or outputs that you want to share?

bazel test --run_under='env XML_OUTPUT_FILE=/tmp/foobar.xml' ..... could be made to a workaround but has major disadvantage of affecting cache and otoh not moving files into location for cached tests (if run_under wrapper creates random output location each time but run_under command itself is the same).

[philwo]

Can you get the data from the Build Event Protocol?

Check our CI script for a naive, unoptimized method how to parse the BEP JSON, get all the test logs and upload them to a server: https://github.com/bazelbuild/continuous-integration/blob/e23e1b90a3d8ce560835e7a5217241b0bbbf9b63/buildkite/bazelci.py#L2013-L2036

You can save the BEP as a JSON file to disk during the build via --build_event_json_file=something.json and maybe add --build_event_json_file_path_conversion=false, too, to prevent Bazel from converting file references to URLs.

[dmivankov]

Will give it a shot, looks like this prints outputs of cached tests too which is good.

Note: --build_event_json_file_path_conversion=false doesn't seem to change anything in json

{
  "id": {
    "testResult": {
      "label": "some_test_label",
      "run": 1,
      "shard": 1,
      "attempt": 1,
      "configuration": {
        "id": "b1f1fcdd5d44b8c93dec65bb7fd1755bd865d799e6714b8fbcca6e57a91e57a0"
      }
    }
  },
  "testResult": {
    "testActionOutput": [
      {
        "name": "test.log",
        "uri": "file:///home/..../test.log"
      },
      {
        "name": "test.xml",
        "uri": "file:///home/...../test.xml"
      }
    ],
    "testAttemptDurationMillis": "400",
    "cachedLocally": true,
    "status": "PASSED",
    "testAttemptStartMillisEpoch": "1632224453667",
    "executionInfo": {}
  }
}

[dmivankov]

With remote cache in place reports are printed as

bytestream://CACHE_ADDRESS/blobs/LONG_HASH/SMALL_NUMBER

like so

  "testResult": {
    "testActionOutput": [
      {
        "name": "test.log",
        "uri": "bytestream://...."
      },
      {
        "name": "test.xml",
        "uri": "bytestream://....."
      }
    ],
    "testAttemptDurationMillis": "3",
    "status": "PASSED",
    "testAttemptStartMillisEpoch": "1632320781462",
    "executionInfo": {
      "strategy": "remote cache hit",
      "cachedRemotely": true
    }
  }

[github-actions]

Thank you for contributing to the Bazel repository! This issue has been marked as stale since it has not had any activity in the last 1+ years. It will be closed in the next 14 days unless any other activity occurs or one of the following labels is added: "not stale", "awaiting-bazeler". Please reach out to the triage team (@bazelbuild/triage) if you think this issue is still relevant or you are interested in getting the issue resolved.

[github-actions]

This issue has been automatically closed due to inactivity. If you're still interested in pursuing this, please reach out to the triage team (@bazelbuild/triage). Thanks!