ci: Add GPU benchmarks and configure with just script

[samuelburnham]

Pull Request Benchmarking

This PR enables two methods for testing GPU performance improvement/regression on future pull requests:

• PR issue_comment: Write !gpu-benchmark in a PR comment to output a comparative fibonacci_lem benchmark with the base branch using boa-dev/criterion-compare. This gives an early signal when performance changes seem likely, but isn't required.
  • Note: this does not yet support forked PRs, see ci: Enable issue_comment actions on forked PRs #816
• Merge Queue (required): Queue a PR to run a comparative benchmark with the base branch, which will check for fibonacci_lem performance regressions using a configurable noise threshold (e.g. 5%) and dequeue the PR if a regression is found. This action outputs the PR bench result as a nu11ptr/criterion-table, which is saved as a persistent comment on the commit (find it by clicking on a commit and scrolling to the bottom of the page).

Benchmark Reports

When a PR is merged to master, a benchmark report for the new commit is deployed to lurk-lab.github.io for historical comparison and user-friendly viewing.

Configuration

The GPU benchmarks are run via a just script, which allows for benchmark configuration with environment variables in CI. This PR adds support for the LURK_RC and LURK_BENCH_NOISE_THRESHOLD variables, which control the reduction counts and noise threshold used in the fibonacci_lem benchmark.

[huitseeker]

We should switch to fibonacci with a canned config, or more specifically pushing a justfile to the repo:
https://github.com/casey/just

[huitseeker]

This is going in the right direction. Hope the comments help!

[samuelburnham]

just is amazing, but thought I'd mention a few issues I've had from a first pass with it:

• No explicitly supported lists or for loops, have to use shell script
• Unclear what the difference is between a recipe, script, and variable
• Comments aren't always comments, e.g. the line before a shebang script
• Conflicting syntax/patterns for global set shell vs shebang script
• No actively maintained VSCode integration
• Beware of hidden tabs when copy-pasting, causing whitespace errors
• Unhelpful error messages make it tricky to debug
• Not great support for nushell
• Doesn't support interpolation of strings and backticks (yet)

None of these are disqualifying, but it will need some additional effort if we want the whole team to use it for bench/CI configuration.

[huitseeker]

This will need a rebase on #808.

For the record, the thing I'm attached to is bench command being separate (in a different file) from github actions / CI workflows. If you would prefer a python script, a shell script, or anything else to host the benchmark command instead of a justfile @samuelburnham, I will be happy to go with your decision: you own this one.

[samuelburnham]

If you would prefer a python script, a shell script, or anything else to host the benchmark command instead of a justfile

Nothing better immediately comes to mind, pardon the late-night rant 😄 In the interests of time LGTM, and advocate for justfile usage mainly in GH actions rather than general-purpose bench scripting for now.

[huitseeker]

Thank you so much, that's a huge step!