Benchmark pull requests against main

[pnmadelaine]

This PR adds the --compare flag to the ./mach benchmark subcommand, allowing to benchmark against another revision of hacl-packages.
It also adds a github action that will use that command on PRs and hacl-star branches, comparing them with main.

[duesee]

Thanks, @pnmadelaine! This looks all very good to me. I have a few (non-blocking) suggestions and a more general question.

The more general question is: I'm sure you tested this locally, but it's difficult for me to see if everything works as expected. Do you think we could extract the benchmarking step into a standalone script that we can use to test locally?

[coveralls]

Pull Request Test Coverage Report for Build 3694501943

• 0 of 0 changed or added relevant lines in 0 files are covered.
• No unchanged relevant lines lost coverage.
• Overall coverage remained the same at 55.966%

---

Totals
Change from base Build 3687385504:	0.0%
Covered Lines:	33754
Relevant Lines:	60312

---

💛 - Coveralls

[pnmadelaine]

@duesee we could definitely extract the script, maybe even put it in mach?

[duesee]

@duesee we could definitely extract the script, maybe even put it in mach?

Good idea 👍

[duesee]

As discussed offline, this is good to go, thanks! I think it would be worth it to try to get rid of the extra download of gbenchmark, though. But it should be fine to leave that as is should there be unexpected complications when changing that.

[pnmadelaine]

Since CI was blocking and I was done moving the script to mach I decided to push everything at once.
The benchmark_pull_request github workflow is now failing because it tries to use the --compare flag from the main branch...
EDIT: I fixed this by changing the behaviour of the --compare flag, it is now meant to be run from the new revision, instead of the old one. This seems more logical anyway.

[duesee]

I'll finish the pending review with "Comment". Feel free to re-request a review when this is ready!

[duesee]

Thank you for the extra effort! This is easier to review now. I checked that mach benchmark --compare works as expected and detects performance regressions. mach returns an error code in this case so that GitHub Actions will (hopefully) fail. Nice work!

[duesee]

@franziskuskiefer, do you want to have another look? In any case we should not forget to squash before merge :-)

[franziskuskiefer]

Thanks, lgtm!