Skip to content

Commit 5a873d3

Browse files
committed
[libc++] Improve error resilience when running historical benchmarks
In benchmark-historical, don't skip gathering the results when the lit command has failed. Indeed, it is expected to fail as part of normal operation since it should be pretty frequent for at least one test to fail when running historical benchmarks. Instead, gather whatever results we have. Also, output the build log in spec.gen.py upon failure so we can see the reason for the failure.
1 parent 48f804d commit 5a873d3

File tree

2 files changed

+2
-5
lines changed

2 files changed

+2
-5
lines changed

libcxx/test/benchmarks/spec.gen.py

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -74,5 +74,5 @@
7474

7575
# Parse the results into a LNT-compatible format. This also errors out if there are no CSV files, which
7676
# means that the benchmark didn't run properly (the `runcpu` command above never reports a failure).
77-
print(f'RUN: %{{libcxx-dir}}/utils/parse-spec-results %T/result/*.train.csv --output-format=lnt > %T/results.lnt')
77+
print(f'RUN: %{{libcxx-dir}}/utils/parse-spec-results %T/result/*.train.csv --output-format=lnt > %T/results.lnt || ! cat %T/result/*.log')
7878
print(f'RUN: cat %T/results.lnt')

libcxx/utils/benchmark-historical

Lines changed: 1 addition & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -90,10 +90,7 @@ def main(argv):
9090
logging.info(f'Running {pretty}')
9191
continue
9292

93-
if subprocess.call(test_cmd) != 0:
94-
logging.error(f'Failed to run the tests for commit {commit}, skipping')
95-
continue
96-
93+
subprocess.call(test_cmd)
9794
output_file.parent.mkdir(parents=True, exist_ok=True)
9895
consolidate_cmd = [(PARENT_DIR / 'consolidate-benchmarks'), build_dir, '--output', output_file]
9996
subprocess.check_call(consolidate_cmd)

0 commit comments

Comments
 (0)