Upload multiple benchmark files in one run?

[larseggert]

Is it possible to upload multiple (hyperfine) results file in with bencher run command, so they show up in a single PR comment?

We are doing this
https://github.com/mozilla/neqo/blob/67ad82d7ae7a72e67034ab285854522918a8b0af/.github/workflows/bencher-upload.yml#L73-L86
at the moment, which causes the PR comment to be overwritten during every loop iteration, which is not great.

[epompeii]

Hey @larseggert thank you for using Bencher, and I'm sorry you're running into trouble.

Is it possible to upload multiple (hyperfine) results file in with bencher run command, so they show up in a single PR comment?

This is currently not possible out of the box. Some folks have hand rolled their own JSON merge scripts to handle this.

With that said though, I would be open extending the --file option so that it can be specified multiple times. This would allow you to upload multiple results files at once. I'm very open to a PR to add this functionality!

We are doing this at the moment, which causes the PR comment to be overwritten during every loop iteration, which is not great.

Ah! In the meantime, you need to specify the --ci-id option to disambiguate the different PR comments. So something like <branch>-<testbed>-<adapter>-<hyperfine benchmark group> would probably do the trick. This will create a separate comment for each benchmark group/iteration of the loop. So long term, it would probably be best to go with one of the above, single comment solutions.

[larseggert]

Thanks for the reply!

Some folks have hand rolled their own JSON merge scripts to handle this.

Would you have a pointer to one of these? I'll look into the --ci-id option in the meantime, but having ten or so individual PR comments will be noisy.

[epompeii]

Would you have a pointer to one of these?

I don't have one handy. You could just reuse the Hyperfine benchmark result type definition, which is what we did in our parser:

bencher/lib/bencher_adapter/src/adapters/shell/hyperfine.rs

Line 22 in aa89713

// https://github.com/sharkdp/hyperfine/blob/ef4049f8f897d4adc4c47a07e60e39d9760fb9ed/src/benchmark/benchmark_result.rs#L11

For each of your benchmark runs, you just add it to the Vec of results.

[epompeii]

@larseggert I have improved the --file option. As of the v0.5.5 release, you can now specify the --file option multiple times. This should solve your issue, no workarounds required.

From the updated docs:

🐰 Tips for using the --file option

The --file option can be used to read benchmark results from a file. You can specify the --file option multiple times to read from multiple files. If you do specify the --file option multiple times, then each file will be considered a separate iteration of the benchmark command. When displayed, these results will be shown in separate tables.

If you wish to combined these results into a single iteration, then you can set the --iter option to 1 and set the --fold option to either the min, max, or median aggregate functions. This will fold the results into a single iteration and display them in a single table. Do not set the --fold option to mean.

---

Note that the "separate tables" would still appear in a single PR comment. So if you want to be able to tell which benchmark results came from which files, then you may want to keep the tables separate. If you don't care, then go on ahead and --fold them down into a single table.

Let me know what you think! More than happy to keep iterating, if there is anything else that you need here.

[larseggert]

Oh thank you!

One question: How does this intersect with --ci-only-on-alert? Will a comment be posted if one of the uploaded benches raises an alert, or do all of them need to do so before a comment is generated?

[epompeii]

How does this intersect with --ci-only-on-alert? Will a comment be posted if one of the uploaded benches raises an alert, or do all of them need to do so before a comment is generated?

Great question! When using the --ci-only-on-alert flag, results get posted as a comment on your PR if any Alerts are generated for that Report. That is, a single Alert is enough to trigger a PR comment.

If an Alert is generated, then all follow up results will also be posted even if they don’t contain any Alerts.

Using the --file option multiple times doesn't change much. It just combines multiple sets of results into a single Report.