Add experimental support for perf (via LinuxPerf.jl)

[Zentrik]

Replaces #325 (closes #325)

Depends on JuliaPerf/LinuxPerf.jl#38.

This pr adds the ability to run perf at the end of a benchmark run for at least a microsecond. It's disabled by default. If enabled it stores the results (adds them to TrialEstimate) with the rest of the benchmark data.

The default is to only measure instructions and branch-instructions as these seem to be the only things non noisy enough to give stable results for the short runs we do. Nor do we need to measure cycles or task-clock given we are running a benchmark beforehand anyways.

For comparison benchmarking BaseBenchmarks.SUITE[["inference", "allinference", "many_local_vars"]] the percentage difference between the max and min minimum time over 5 runs is 1.25% whilst it's 0.002% for instruction count. Given on Nanosoldier the same benchmark is changing about 5-10% every few runs (see here), this seems like it's going to work very well

[DilumAluthge]

I haven't added tests, but given the CI doesn't have perf available, not sure how useful it would be.

We might need to set up Buildkite CI on this repo. @vchuravy @staticfloat

[willow-ahrens]

I'm not sure I have the bandwidth as a maintainer of this package to support this kind of infrastructure directly in benchmarktools.jl. Is there a way that we can support this inside LinuxPerf.jl rather than in BenchmarkTools.jl? Maybe a separate run_perf function that runs the benchmarks in perf?

[willow-ahrens]

it seems like too much overhead to me to support separate CI for this just to support this feature here, especially if it's possible to support this in linuxperf.jl. Could linuxperf support a separate run_perf function that runs a BenchmarkTools Benchmark through perf?

Especially if we need special CI to run some software, it seems much easier to support this kind of feature in that repo, rather than this one.

[Zentrik]

I think if I can get this to a low enough maintenance overhead for you it would be best to add perf profiling in BenchmarkTools. I think we can just remove the extra CI and make it clear that this is experimental. I don't think there's much need to make sure the profiling works on CI and users can easily run their own tests. Let me know if there's anything else I can do to lower the overhead or if it's not going to be possible for the overhead to be low enough.

If you care why I don't think it makes sense to put this functionality in LinuxPerf or BaseBenchmarks, I've explained below.
So running perf on samplefunc seems to work ok, it has fairly high overhead but it seems to be not much noisier than this pr which is what really matters. However, actually integrating this in LinuxPerf or BaseBenchmarks seems difficult as presumably they would have to overload BenchmarkTools._run so that they can run perf whilst the function is still hot when benchmarking a BenchmarkGroup. I don't think we would want to do this by default for people using LinuxPerf so if I was to do this I would probably put it in BaseBenchmarks which doesn't have CI anyways. Also, they would have to setup a way to save and load the perf profiling results.

[willow-ahrens]

Could we add something like a prehook and a posthook argument to the run function, so that other packages can export their own versions of run with special behavior, such as bprofilerun, bprofileviewrun, bperfrun? Is there a way to make them nest?

[willow-ahrens]

I'm imagining something like:

run_perf(args...; kwargs..., posthook=f) = run(args...; kwargs..., posthook=(b)->(run_perf_stuff(b); f(b)))