Continuous benchmarks and profiling

[charithe]

• Benchmark on every pull request
• Profile before each release

https://github.com/rhysd/github-action-benchmark seems unmaintained and is unable to parse a suite of benchmarks. Need to find a new way to do continuous benchmarks.

[charithe]

https://github.com/vitessio/arewefastyet

[isdevx]

It seems like https://github.com/rhysd/github-action-benchmark showed some pulse recently. I tested the benchmarks on my own fork and was able to run it. I wonder if we should add this right away as it's pretty straightforward to add. We can experiment it for a while if it makes sense. cc @charithe

[charithe]

SGTM. If they have fixed the issue with parsing Go benchmark suites, it would be good to incorporate it into the PR workflow.

[isdevx]

Oh that escaped from me 🤦 Nonetheless, I forked the action and used a fork that includes the fix for parsing issue. Now I pointed to custom version and it works fine. We can go with that version until it gets fixed on the upstream.

[charithe]

Are they open to accepting a PR to fix the issue? I would prefer to use the "official" action if possible.

[isdevx]

There is already a PR for the issue benchmark-action/github-action-benchmark#32 and I forked that fork. I'd say they are open for contributions but there are many to review and consolidate. While waiting for the resolution, we can start using and experimenting the action. The transition between official action and my fork is one line change.

[charithe]

That PR has been open for over a year so it's unlikely that it will get merged soon. I wonder whether there are other alternatives -- even a more actively maintained fork of this one -- that we can use.

Our own fork should be the last resort. If we do decide to do it, let's fork it to the cerbos org.

[charithe]

https://bencher.dev