BENCH: update asv, fail benchmark CI on errors, fix fallout

[h-vetinari]

Rebased #18857 + trying to debug.

Towards #18860 (done in #19104)
Closes #18868
Closes #18849

[h-vetinari]

Hey all, the benchmarks are currently broken (as in: several of them fail silently), and have been for a good two years at least (that's when wminkowski was removed, it was still being called in the benchmarks).

I've tried to skip the failures here (with the exception of those added in 3c54130, I think they never passed. I asked the author to take a look in #18860, but haven't heard back yet). I don't expect this to be mergeable as is, but wanted to discuss how we can get to a point where we can unbreak this.

BTW, this also solves the current asv-pin. Key insight (from @HaoZeke in #18857, which this PR rebases) is that we need to pass --quick to get the pre-0.5 behaviour. There've been other asv improvements; currently this uses a development version, but a new asv-release is apparently not too far away according to @HaoZeke,

[j-bowhay]

I am happy with skipping and raising an issue to track (most would probably make fairly good first contributions). Clearly, if these have been broken for so long I don't think they need fixing instantly however it would be good to get this in to prevent the situation from getting worse.

[h-vetinari]

@HaoZeke, are you fine with the compressed form of #18857 here? Then I think we'd be ready to fix this issue.

@mdhaber @tupui @rgommers feedback welcome, otherwise I'm planning to merge this without much further changes (and would open a follow-up issue for fixing the failing-and-now-skipped benchmarks).

[HaoZeke]

@HaoZeke, are you fine with the compressed form of #18857 here? Then I think we'd be ready to fix this issue.

Yup, thanks for seeing this through 😊