Revive performance regression testing?

[johnmyleswhite]

As discussed in #11683, it would be great to revive some kind of performance regression testing so that we don't slowly accumulate perf problems.

In an ideal world, we'd standardize on a system that non-Base packages could use as well. But for now it would be good to just get Base being tested for perf rigorously again.

[carnaval]

Here are some things about my dream benchmark suite (maybe too much work, but one can hope) :

• Not something you run on one commit and gives you number, but something you run on two commits and reports a comparison.
• Something that does not compare numbers from last week with numbers from today. Ideally when you ask to compare two commits, it runs both benchs right away. This avoids the "oh well yeah it seems slower but we changed the RAM on the server yesterday".
• I'm a total idiot concerning statistics but I'm pretty sure there are some knowledgeable people in this crowd : I would like something with significance tests, both for single benchmarks (e.g. is the variability low enough compared to the measure, but fancier ?) and for the comparison itself (is the reported inc/decrease significant).

To summarize, we need to have something very robust that we can trust. I don't want to have what we can see right now with everyone handwaving "unrelated" CI crashes which then makes us introduce actual bugs that should have been caught. It is starting to irritate me a bit :-)

[yuyichao]

@carnaval What about The Phoronix Test Suite I've never use it myself but it seems that it is pretty good at comparing two things (commits) and they are pretty serious about it (so they are probably doing the statistics right).

A timeline like what codespeed provide would still be nice though.

[KristofferC]

Couldn't a first step be to simply revive http://speed.julialang.org? Was the problem a lack of finances for hardware or was there significant manual work needed keeping the site up? I think it is easier to get people to write performance test if they have some pretty graphs/numbers to look at. Getting something decent going is good enough while being on the way to the "performance testing nirvana" @carnaval mentioned.

[johnmyleswhite]

All great points. My take on each issue:

• The ideal perf monitoring system needs to do both time series analysis and binary comparisons. Binary comparisons are really valuable (and they make the statistical design much, much easier), but they suffer from a failure mode that's, in my experience, very common in practice: the gradual accumulation of small perf regressions that aren't easily detectable individually, but are detectable after say 100 or 1000 commits get merged. You, of course, can detect these cumulative regressions if you make binary comparisons over longer periods of time, but they're often both more obvious and more informative when you see them as a trend over time.
• Comparisons across system changes are, as you say, very brittle. In my experience, you can detect things like server hardware changes by running post-hoc A/A tests -- in particular, by rerunning one of last week's commits to confirm that the measurement system is still stable. You also tend to notice such changes from time series graphs, since you see clear trend breaks when the underlying system changes. In general, my preference is to do extensive logging of the measurement system's state since you can often use modeling techniques to correct for these sorts of problems. As your measurement system becomes more complex (e.g. running benchmarks on a fleet of machines), you have to do modeling -- so it's good to do it from the start.
• The statistics of one-commit benchmarks are handled extremely well by Bryan O'Sullivan's Criterion library, which we should port to Julia. (Jake already started doing this in the past.) For binary comparisons, the analysis is pretty easy -- unless you want to report comparisons in terms of percentage changes. If you do the latter (which is really helpful for reasoning about directionally-inconsistent changes across multiple benchmarks), you need to use something like Fieller's theorem to generate useful summaries of your comparisons because the non-parametric bootstrap that Criterion employs will generate mildly invalid confidence intervals for percentage changes.
• As you say, minimizing false positives is by far the most important thing about a testing system. It's much better to have a regular false negative problem than to teach people that the testing system can't be trusted.

[tkelman]

+1 for porting Criterion, have heard nothing but praise for that library.

Based on @KristofferC's comments here #11683 (comment) I will retract my suggestion of looking into CDash, it looked like it could save some work in just having a place to put large amounts of historical test data, but if it's a Jenkins-style quagmire of configuration files to set up then nevermind. Also looks like they only store 30 days worth of data on free plans.

This jogged my memory, there was a past issue on this at #9456 which was closed without any particular resolution.

[johnmyleswhite]

I've got about half of the work done to get something inspired by Criterion finished. Should have final results tomorrow night.

[IainNZ]

Speed.julialang.org was @staticfloat's baby. At the time we didn't have the build infrastructure we have now, so I think it because it was trying to run on every commit and was also dealing with build issues there were a lot of problems. Many of the perf tests didn't seem to to be set up to run long enough as well, creating some spurious results. It'd probably be easier to get something working now (using e.g. generic nightly linux binaries, like PackageEvaluator does)

[IainNZ]

Would love to write some perf tests too, especially for natural-but-not-highest-possible-perf code (the kind you see on Stackoverflow). Only worth it if we're tracking them.

[tkelman]

On the infrastructure side of plumbing up the github webhooks, we should probably borrow some tools from Rust and use https://github.com/barosl/homu to help address JuliaCI/julia-buildbot#1. The buildbots we have at http://buildbot.e.ip.saba.us:8010/builders run surpringly often on master. To start with we should work out a method of pinging a @juliabot (looks like that's taken but inactive, maybe @juliabuild?) in the comments of a PR to fire off perf and/or PkgEvaluator runs on-demand.

[IainNZ]

IIRC @Keno owns @Juliabot

[IainNZ]

And I was messing around with this: https://github.com/MechaJulia/MechaJulia for a while

[Keno]

I believe I have @julia-bot with a dash.

[tkelman]

MechaJulia's not a bad name, but dash vs no dash would get misspelled way too often.

Realistically buildbot and other projects have already done a lot of work here, there are plenty of off-the-shelf solutions we can start from. The inspired-by-criterion benchmarking harness should probably be written in Julia, but most of the other infrastructure shouldn't IMO.