Set benchmark runs to error if the subprocess errors (#5434)

* Set benchmark runs to error if the subprocess errors.

* Still compare results even from a broken run.

* Still upload reports if overnight run fails.

* What's New entry.

.github/workflows/benchmarks_run.yml

@@ -112,6 +112,7 @@ jobs:
           gh issue create --title "$title" --body "$body" --label "Bot" --label "Type: Performance" --repo $GITHUB_REPOSITORY
 
       - name: Upload any benchmark reports
+        if: success() || steps.overnight.outcome == 'failure'
         uses: actions/upload-artifact@v3
         with:
           name: benchmark_reports

benchmarks/bm_runner.py

@@ -51,6 +51,7 @@ def _subprocess_runner(args, asv=False, **kwargs):
         args.insert(0, "asv")
         kwargs["cwd"] = BENCHMARKS_DIR
     echo(" ".join(args))
+    kwargs.setdefault("check", True)
     return subprocess.run(args, **kwargs)
 
 
@@ -387,16 +388,20 @@ def func(args: argparse.Namespace) -> None:
         _setup_common()
 
         commit_range = f"{args.first_commit}^^.."
-        asv_command = shlex.split(ASV_HARNESS.format(posargs=commit_range))
-        _subprocess_runner([*asv_command, *args.asv_args], asv=True)
-
         # git rev-list --first-parent is the command ASV uses.
         git_command = shlex.split(
             f"git rev-list --first-parent {commit_range}"
         )
         commit_string = _subprocess_runner_capture(git_command)
         commit_list = commit_string.split("\n")
-        _asv_compare(*reversed(commit_list), overnight_mode=True)
+
+        asv_command = shlex.split(ASV_HARNESS.format(posargs=commit_range))
+        try:
+            _subprocess_runner([*asv_command, *args.asv_args], asv=True)
+        finally:
+            # Designed for long running - want to compare/post any valid
+            #  results even if some are broken.
+            _asv_compare(*reversed(commit_list), overnight_mode=True)
 
 
 class Branch(_SubParserGenerator):

docs/src/whatsnew/latest.rst

@@ -93,6 +93,9 @@ This document explains the changes made to Iris for this release
    an issue if the overnight run fails - this was previously an 'invisible'
    problem. (feature branch: :pull:`5432`)
 
+#. `@trexfeathers`_ set `bm_runner.py` to error when the called processes
+   error. This fixes an oversight introduced in :pull:`5215`. (feature branch:
+   :pull`5434`)
 
 .. comment
     Whatsnew author names (@github name) in alphabetical order. Note that,